Image splicing method and device, medium and product

Abstract

The invention provides an image splicing method and device, a medium and a product. The method comprises the following steps: respectively determining average pixel values of target areas in a plurality of original images; the target region is a region corresponding to a splicing region in a finally generated target aerial view in the original image; the splicing area is a whole overlapping area or a part overlapping area between the top view images in the target aerial view; determining a compensation coefficient corresponding to each original image according to the average pixel value of each target area and a preset constraint equation set; and performing compensation processing and splicing conversion processing on each original image according to the compensation coefficient to generate a target aerial view. According to the image splicing method, the compensation coefficient generated based on the average pixel value of the target area can more effectively adjust the color difference between the original images, so that the color difference of the finally generated target aerial view is smaller, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image splicing method, device, medium and product. BACKGROUND

[0002] With the increasing intelligence of the automobile industry and the decreasing cost of vehicle-mounted cameras, multiple dispersed images can be spliced to generate an overall bird's-eye view of the virtual view angle of the roof by mounting multiple fisheye cameras. Through the bird's-eye view, the driver can more clearly identify the obstacles around the vehicle, providing convenience for driving, parking, etc.

[0003] Since different fisheye cameras are generally installed around the vehicle body, the light intensity of different cameras often differs, especially in dark, partially blocked and fast-moving scenes, the light difference is more obvious, so the original images captured by different fisheye cameras will have obvious brightness and color differences.

[0004] In such a scene, the bird's-eye view spliced based on the original images captured by different fisheye cameras will also present obvious color differences, resulting in poor user experience. SUMMARY

[0005] The present application provides an image splicing method, device, medium and product to solve the problem that the bird's-eye view spliced based on the original images captured by different fisheye cameras will present obvious color differences, resulting in poor user experience.

[0006] The first aspect of the present application provides an image splicing method, comprising:

[0007] determining the average pixel value of a target region in each of the multiple original images; the target region is a region in the original image corresponding to a splicing region in the finally generated target bird's-eye view; the splicing region is all or part of the overlapping region between each overhead image in the target bird's-eye view;

[0008] determining a compensation coefficient corresponding to each original image according to the average pixel value of each target region and a preset constraint equation set;

[0009] performing compensation processing and splicing conversion processing on each original image according to the compensation coefficient to generate a target bird's-eye view.

[0010] Further, the method described above, the determining the average pixel value of a target region in each of the multiple original images, comprises:

[0011] converting the pixels of each target region to an RGB color space; the pixels of the target region are determined based on a preset mapping table and the pixels of the splicing region in the target bird's-eye view;

[0012] Calculate average pixel values corresponding to RGB color channels in the target region.

[0013] Further, the method as described above, the compensation coefficient corresponding to each original image is determined according to the average pixel value of each target region and a preset constraint equation group, comprising:

[0014] For each target region, the following processing is performed:

[0015] The average pixel value corresponding to each color channel is input into the corresponding preset constraint equation group to construct a target constraint equation group.

[0016] Solve the target constraint equation group to generate compensation coefficients in the corresponding color channel of each original image; the absolute value of the compensation coefficient is less than a preset compensation threshold.

[0017] Further, the method as described above, the compensation coefficient corresponding to each original image is determined according to the average pixel value of each target region and a preset constraint equation group, comprising:

[0018] If the absolute value of the function value of the target constraint equation group is greater than a preset color difference threshold, the compensation coefficient in the corresponding color channel of the largest original image is set to a preset value; the largest original image is the original image to which the target region with the largest average pixel value belongs.

[0019] Solve the target constraint equation group to generate compensation coefficients in the corresponding color channel of other original images; the other original images are other original images except the largest original image.

[0020] If the absolute value of the function value of the target constraint equation group is less than or equal to a preset color difference threshold, the target constraint equation group is solved to generate compensation coefficients in the corresponding color channel of each original image.

[0021] Further, the method as described above, the splicing region in the target bird's eye view is a region formed by a new boundary line and a boundary line of the target bird's eye view after the intersection line in the target bird's eye view is rotated by a preset center angle based on the center; the new boundary line is the intersection line after being rotated by the preset center angle.

[0022] The compensation processing and splicing conversion processing of each original image according to the compensation coefficient to generate a target bird's eye view, comprising:

[0023] Compensate the target region according to the first angle, the preset center angle, and the compensation coefficient to generate a compensated target region; the first angle is the angle between the connecting line formed by each pixel in the splicing region and the center and the new boundary line.

[0024] According to the compensation coefficient, a remaining area in each original image is compensated to generate a compensated remaining area; the remaining area is an area in the original image except the target area;

[0025] The compensated target area and the compensated remaining area are stitched and converted to generate a target bird's eye view.

[0026] Further, the method described above, the compensation of the target area according to the first included angle, the preset central angle of the circle and the compensation coefficient, the compensated target area includes:

[0027] According to the first included angle and the preset central angle of the circle, the stitching weight of each pixel in the target area is determined;

[0028] For each pixel in the target area, the stitching weight, the compensation coefficient and the pixel value of the corresponding pixel in the target area are weighted to generate the pixel value of the corresponding pixel in the compensated target area;

[0029] Based on the pixel value of each pixel in the compensated target area, the compensated target area is generated.

[0030] Further, the method described above, the new boundary line has a corresponding relationship with the original image;

[0031] The stitching weight of each pixel in the target area is determined according to the first included angle and the preset central angle of the circle, including:

[0032] According to the corresponding relationship, the second included angle is determined from the first included angle; the new boundary line of the second included angle corresponds to the original image to which the target area belongs;

[0033] According to the quotient between the second included angle and twice the preset central angle of the circle, the corresponding stitching weight of each pixel in the target area is determined.

[0034] Further, the method described above, before determining the average pixel value of the target area in each of the plurality of original images, further includes:

[0035] The calibration parameters corresponding to the historical original image and the historical bird's eye view are obtained;

[0036] Based on the calibration parameters, the target area of the historical original image is inversely calculated from the stitching area of the historical bird's eye view;

[0037] The pixel mapping relationship between the stitching area and the target area is stored in a preset mapping table.

[0038] The second aspect of the present application provides an image stitching device, including:

[0039] The first determining module is configured to determine average pixel values of target regions in the plurality of original images respectively; the target regions are regions in the original images corresponding to a splicing region in the finally generated target bird's eye view; and the splicing region is all or part of overlapping regions between the overhead images in the target bird's eye view.

[0040] The second determining module is configured to determine compensation coefficients corresponding to the original images according to the average pixel values of the target regions and a preset constraint equation set.

[0041] The generating module is configured to perform compensation processing and splicing conversion processing on the original images according to the compensation coefficients, and generate the target bird's eye view.

[0042] Further, the first determining module is specifically configured to:

[0043] convert the pixels of the target regions to an RGB color space; the pixels of the target regions are determined based on a preset mapping table and pixels of the splicing region in the target bird's eye view; and the average pixel values corresponding to the RGB color channels in the target regions are calculated.

[0044] Further, the second determining module is specifically configured to:

[0045] For each target region, the following processing is performed:

[0046] the average pixel values corresponding to the color channels are input into a corresponding preset constraint equation set to construct a target constraint equation set; and the target constraint equation set is solved to generate compensation coefficients of the corresponding color channels in the original images; and the absolute value of the compensation coefficients is less than a preset compensation threshold.

[0047] Further, when the second determining module solves the target constraint equation set to generate the compensation coefficients of the corresponding color channels in the original images, the second determining module is specifically configured to:

[0048] if the absolute value of a function value of the target constraint equation set is greater than a preset color difference threshold, the compensation coefficient of the corresponding color channel in the largest original image is set to a preset value; the largest original image is an original image to which a target region with the largest average pixel value belongs; the target constraint equation set is solved to generate the compensation coefficients of the corresponding color channels in other original images; the other original images are original images other than the largest original image; and if the absolute value of the function value of the target constraint equation set is less than or equal to the preset color difference threshold, the target constraint equation set is solved to generate the compensation coefficients of the corresponding color channels in the original images.

[0049] Further, the device, the splicing area in the target bird's eye view is an area formed by a new boundary line and a boundary line of the target bird's eye view after an intersection line in the target bird's eye view is rotated by a preset center angle to two sides based on a center; the new boundary line is the intersection line after being rotated by the preset center angle;

[0050] The generating module is specifically configured to:

[0051] compensate the target area according to the first included angle, the preset center angle and the compensation coefficient to generate a compensated target area; the first included angle is an angle between a connecting line between each pixel in the splicing area and the center and the new boundary line; compensate the remaining area in each original image according to the compensation coefficient to generate a compensated remaining area; the remaining area is an area in the original image except the target area; and splice and convert the compensated target area and the compensated remaining area to generate the target bird's eye view.

[0052] Further, the device, the generating module is specifically configured to:

[0053] determine a splicing weight of each pixel in the target area according to the first included angle and the preset center angle; and perform weighted processing on a pixel value of a corresponding pixel in the compensated target area by using the splicing weight, the compensation coefficient and the pixel value of the corresponding pixel in the target area to generate the pixel value of the corresponding pixel in the compensated target area; and generate the compensated target area based on the pixel values of the pixels in the compensated target area.

[0054] Further, the device, the new boundary line has a corresponding relationship with the original image;

[0055] The generating module is specifically configured to:

[0056] determine a second included angle from the first included angle according to the corresponding relationship; the new boundary line of the second included angle corresponds to the original image to which the target area belongs; and determine a corresponding splicing weight of each pixel in the target area according to a quotient between the second included angle and twice the preset center angle.

[0057] Further, the device, the device further comprises:

[0058] a mapping module configured to acquire calibration parameters corresponding to historical original images and historical bird's eye views; inversely calculate a target area of a historical original image from a splicing area of a historical bird's eye view based on the calibration parameters; and store a pixel mapping relationship between the splicing area and the target area in a preset mapping table.

[0059] The third aspect of the present application provides an electronic device, comprising a memory and a processor;

[0060] The memory stores computer-executable instructions;

[0061] The processor executes the computer-executable instructions stored in the memory to implement the image stitching method according to any one of the first aspect.

[0062] The fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the image stitching method according to any one of the first aspect.

[0063] The fifth aspect of the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the image stitching method according to any one of the first aspect.

[0064] The image stitching method, device, medium and product provided by the present application, the method comprises: determining the average pixel value of a target region in a plurality of original images respectively; the target region is a region in the original image corresponding to a stitching region in a finally generated target bird's eye view; the stitching region is all or part of the overlapping region between each overhead image in the target bird's eye view; determining a compensation coefficient corresponding to each original image according to the average pixel value of each target region and a preset constraint equation group; performing compensation processing and stitching conversion processing on each original image according to the compensation coefficient, and generating a target bird's eye view. The image stitching method of the present application, since the target region corresponds to the stitching region of the target bird's eye view, the compensation effect of the compensation coefficient determined based on the average pixel value of the target region on the target region and other regions in the original image is better, so that the color difference between each original image can be adjusted more effectively based on the compensation coefficient, the color difference of the finally generated target bird's eye view is smaller, and the user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0065] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0066] Figure 1a The image stitching flowchart provided by the present application;

[0067] Figure 1b The bird's eye view schematic diagram provided by the present application;

[0068] Figure 2a The fisheye stitching schematic diagram provided by the present application;

[0069] Figure 2b Fig. 1 is a fisheye schematic diagram provided for the present application;

[0070] Figure 3 Fig. 1 is a fisheye schematic diagram provided for the present application;

[0071] Figure 4 Fig. 1 is a fisheye schematic diagram provided for the present application;

[0072] Figure 5 Fig. 1 is a fisheye schematic diagram provided for the present application; Figure Three

[0073] Figure 6 Fig. 1 is a fisheye schematic diagram provided for the present application;

[0074] Figure 7 Fig. 1 is a fisheye schematic diagram provided for the present application;

[0075] Figure 8 Fig. 1 is a fisheye schematic diagram provided for the present application;

[0076] The specific embodiments of the present application have been shown through the above-mentioned drawings, and will be described in more detail hereinafter. These drawings and the written description are not intended to restrict the scope of the present application concept in any way, but to illustrate the present application concept to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0077] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The embodiments described in the following exemplary embodiments are not meant to represent all implementations consistent with the present application. Rather, they are simply examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0078] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0079] In order to clearly understand the technical solutions of the present application, the concept process of the technical solutions will be described in detail first. At present, the image stitching method is as follows Figure 1a ​As shown in the figure, four fisheye cameras, i.e., fisheye camera 1 to fisheye camera 4 in the figure, are used for exemplary illustration. The fisheye camera 1 to fisheye camera 4 after brightness fine-tuning shoot to generate corresponding images, i.e., image 1 to image 4 in the figure. At the same time, the image 1 to image 4 is processed by the camera calibration parameters for the top view splicing to generate a top view splicing image. The top view splicing image is combined with the preset vehicle model for fusion to generate a bird's eye view, specifically as shown in the figure. Figure 1b As shown in the figure, fisheye 1 represents the image area corresponding to the fisheye camera 1, and similarly, other areas also correspond to the corresponding fisheye camera.

[0080] Because different fisheye cameras are generally installed around the vehicle body, the light intensity of different cameras often differs, especially in dark, partially blocked, and fast-moving scenes, the light difference is more obvious, thus the image 1 to image 4 generated by the fisheye camera 1 to fisheye camera 4 may have a large color difference, and the finally generated bird's eye view also has a significant color difference, resulting in poor user experience.

[0081] Therefore, in view of the problem that the bird's eye view generated by splicing the original images shot by different fisheye cameras in the prior art will present a significant color difference, resulting in poor user experience, the inventors found in research that the compensation coefficient of the original image can be determined through the target area corresponding to the bird's eye view in the original image. Because the compensation coefficient is related to the target area, it can reduce the color difference of the target area and further reduce the color difference between the original images, so that the color difference of the finally generated target bird's eye view is smaller, improving the user experience.

[0082] Specifically, the image splicing and generation process is as follows:

[0083] The average pixel value of the target area in the plurality of original images is determined respectively, wherein the target area is the area in the original image corresponding to the splicing area in the finally generated target bird's eye view, and the splicing area is the entire overlapping area or part of the overlapping area between the top view images in the target bird's eye view. The compensation coefficient corresponding to each original image is determined according to the average pixel value of each target area and a preset constraint equation set. The compensation coefficient is used for compensation processing and splicing conversion processing of each original image to generate a target bird's eye view.

[0084] The image splicing method of the present application, because the target area corresponds to the splicing area of the target bird's eye view, the compensation coefficient determined based on the average pixel value of the target area has a better compensation effect on the target area and other areas in the original image, so that based on the compensation coefficient, the color difference between the original images can be adjusted more effectively, so that the color difference of the finally generated target bird's eye view is smaller, improving the user experience.

[0085] Based on the above creative findings, the inventors proposed the technical solution of the present application.

[0086] The application scenario of the image splicing method provided in the embodiments of the present application is introduced as follows. As shown in FIG. 1, the figure exemplarily shows each region in the bird's eye view, which are fish eye 1 corresponding region to fish eye 4 corresponding region, and a car model. Among them, A1, B1, C1 and D1 are the joint regions of the bird's eye view in the image splicing method of the embodiments, the fish eye 1 corresponding region includes A1 and B1, the fish eye 2 corresponding region includes B1 and D1, the fish eye 3 corresponding region includes C1 and D1, and the fish eye 4 corresponding region includes A1 and C1. The car model is an image model representing the host vehicle in the bird's eye view. The embodiments exemplarily use a simple image model to represent the host vehicle. Figure 2a

[0087] Exemplarily, when image splicing is needed, the following process is performed:

[0088] ①The average pixel value of the target region in the original image photographed by fish eye 1 to fish eye 4 is determined respectively. As shown in FIG. 2, the figure exemplarily shows the original image corresponding to fish eye 1, and the shape and size of the target region of the original image are only used for exemplarily showing for convenience of description. The actual shape and size can be different from that in FIG. 1. The embodiments are not limited thereto. The target region A2 in the original image corresponds to the joint region A1 in FIG. 1, and the target region B2 corresponds to the joint region B1 in FIG. 1. Figure 2b Figure 2a Figure 2a

[0089] ②The compensation coefficient corresponding to each original image is determined according to the average pixel value of each target region and the preset constraint equation group.

[0090] ③The compensation processing and splicing conversion processing are performed on each original image according to the compensation coefficient, to generate a target bird's eye view.

[0091] The embodiments of the present application are introduced in combination with the drawings of the specification.

[0092] Figure 3 The flowchart of the image splicing method provided in the present application is shown in FIG. 3. The execution subject of the embodiments of the present application is an image splicing device, which can be integrated in an electronic device, such as a vehicle terminal. The image splicing method provided in the embodiments of the present application includes the following steps: Figure 3

[0093] Step S101, the average pixel value of the target region in the original image is determined respectively. The target region is the region in the original image corresponding to the splicing region in the finally generated target bird's eye view. The splicing region is the entire overlapping region or partial overlapping region between each bird's eye view in the target bird's eye view.

[0094] ​​​​​In this embodiment, the splicing area can be set according to user demand or actual application demand (camera type, vehicle size, actual environmental influence, etc.), for example, only the part with large color difference can be set as the splicing area, so that when image compensation is performed subsequently, the overall efficiency of compensation can be improved by focusing on the compensation of the key area.

[0095] The original image can be obtained from other electronic devices, such as a storage device storing the original image, or from a shooting device, such as a fisheye camera, a binocular camera, etc. The present embodiment does not make any limitation on this. At the same time, since the shooting device is generally distributed around the vehicle and shoots the scene around the vehicle, there can be an overlapping part (common visible area) in the area shot by each shooting device. The area in the overlapping part corresponding to the splicing area in the target bird's eye view is the target area.

[0096] The overlapping part between the original images is related to the position area and the shooting angle of the shooting device. The target area corresponding to the original image can be determined based on the camera internal and external parameters, the position area and the shooting angle of the shooting device. In addition, the target area of the original image can also be determined by inverse calculation based on the splicing area of the bird's eye view and the camera internal and external parameters of the shooting device.

[0097] Optionally, in the present embodiment, before S101, the mapping relationship between the target area in the original image and the splicing area in the bird's eye view can also be determined in advance, so that in S101, the target area can be directly determined according to the mapping relationship, without the need to determine the target area by inverse calculation of the splicing area of the bird's eye view in the current process, thereby improving the efficiency of determining the target area. The specific process is as follows:

[0098] Obtain the calibration parameters corresponding to the historical original image and the historical bird's eye view.

[0099] Inverse calculate the target area of the historical original image from the splicing area of the historical bird's eye view based on the calibration parameters.

[0100] Store the pixel mapping relationship between the splicing area and the target area in a preset mapping table.

[0101] In this embodiment, when the original image is converted into the bird's eye view, the image conversion is performed based on the corresponding calibration parameters. The calibration parameters include camera intrinsic parameters, camera extrinsic parameters and distortion parameters, which can be obtained by calibration methods such as linear calibration and nonlinear calibration. Thus, the target region of the historical original image can be inversely calculated from the splicing region in the historical bird's eye view through the calibration parameters. By storing the pixel mapping relationship between the splicing region and the target region in the preset mapping table, when the target region in the new original image needs to be determined in the future, the pixel mapping relationship stored in the preset mapping table can be directly determined, which improves the efficiency of determining the target region, and further improves the color optimization and image generation efficiency of the bird's eye view.

[0102] In step S102, the compensation coefficients corresponding to each original image are determined according to the average pixel value of each target region and the preset constraint equation set.

[0103] In this embodiment, the average pixel value is the average value of all pixels in a single target region. The average pixel value of each target region can be used to determine the compensation coefficient corresponding to each original image by solving the preset constraint equation set. The preset constraint equation set is an equation set with compensation coefficients and average pixel values as variables, and the function value of the preset constraint equation set satisfies a certain constraint condition. The compensation coefficient is a coefficient used to calculate the pixel value of the compensated original image when performing color compensation processing on the original image.

[0104] In order to prevent over-compensation from causing image color distortion, a threshold value of the compensation coefficient can be set to limit the compensation amount, so that the compensation coefficient obtained by solving is less than the threshold value of the compensation coefficient, thereby avoiding over-compensation of the original image. At the same time, in order to deal with the problem of excessive color difference, a color difference limit threshold value can also be set. If the function value of the preset constraint equation set is greater than the color difference limit threshold value, the compensation coefficients can be processed in a certain way, such as setting a certain compensation coefficient to a fixed value, and based on the fixed value, other compensation coefficients are solved, thereby solving the problem of excessive color difference.

[0105] In step S103, the original images are compensated and spliced and converted according to the compensation coefficients, and the target bird's eye view is generated.

[0106] In this embodiment, according to the compensation coefficient, the target region and other regions in each original image can be compensated correspondingly. For example, the pixel value of the original image can be directly multiplied by the corresponding compensation coefficient to obtain the pixel value of the compensated original image. Alternatively, the compensation coefficients can be processed correspondingly according to the region type of the original image to generate optimized compensation coefficients. For example, the compensation coefficients of two original images with overlapping parts can be fused by weighting to generate fused compensation coefficients. Meanwhile, the target region is compensated by using the fused compensation coefficients, and the specific manner can be the multiplication method described above to obtain the pixel value of the compensated target region. For other regions, the corresponding compensation can be performed according to the compensation coefficient of the original image to generate the compensated other regions.

[0107] Meanwhile, the compensated original image is converted into an overhead view, and the overhead views are spliced to generate the corresponding target bird's eye view.

[0108] The image splicing method provided in the embodiments of the present application includes the following steps.

[0109] The image splicing method provided in the embodiments of the present application includes the following steps.

[0110] Figure 4 The flowchart of the image splicing method provided in the embodiments of the present application is shown in FIG. 2. Figure 4 The image splicing method provided in the embodiments of the present application is based on the image splicing method provided in the previous embodiment of the present application and is further refined. The image splicing method provided in the embodiments of the present application includes the following steps.

[0111] In step S201, the pixels of each target region are converted to the RGB color space. The pixels of the target region are determined based on a preset mapping table and the pixels of the splicing region in the target bird's eye view. The mapping relationship between the pixels of the target region and the pixels of the splicing region is recorded in the preset mapping table, and the pixels of the corresponding target region can be queried from the preset mapping table based on the pixels of the splicing region.

[0112] The video stream of the fisheye camera commonly used at present is in YUV color space, containing brightness and chrominance, and is suitable for encoding and storage. In the embodiment, the RGB color space is suitable for image acquisition and display, and by converting the pixels of the target region from other color spaces (such as YUV color space) to the RGB color space, a basis can be provided for subsequent corresponding compensation of each color channel, thereby improving the color compensation effect.

[0113] In step S202, the average pixel value corresponding to the RGB color channel in the target region is calculated.

[0114] In the embodiment, the average pixel value corresponding to each color channel in the RGB color channel is calculated, that is, the average pixel value corresponding to the R color channel, the G color channel and the B color channel exists.

[0115] In step S203, for each target region, the following processing is performed:

[0116] In step S204, the average pixel value corresponding to each color channel is input into the corresponding preset constraint equation set to construct a target constraint equation set.

[0117] In the embodiment, if the original images are four, that is, image 1, image 2, image 3 and image 4, and the corresponding vehicle regions are front, right, rear and left, respectively, the preset constraint equation set can be set as follows:

[0118]

[0119] wherein, G f is the function value corresponding to the G color channel of the constraint equation set, A1, B1, B2, D2, C3, D3, A4 and C4 are the average pixel values corresponding to the target regions, wherein A1 and A4 are the average pixel values of the target regions corresponding to the joint regions between image 1 and image 4, B1 and B2 are the average pixel values of the target regions corresponding to the joint regions between image 1 and image 2, C3 and C4 are the average pixel values of the target regions corresponding to the joint regions between image 3 and image 4, and D2 and D3 are the average pixel values of the target regions corresponding to the joint regions between image 2 and image 3. f1 is the compensation coefficient corresponding to image 1, f2 is the compensation coefficient corresponding to image 2, f3 is the compensation coefficient corresponding to image 3, and f4 is the compensation coefficient corresponding to image 4.

[0120] Similarly, the R channel and the B channel can also be set similarly, which will not be described here.

[0121] In step S205, the target constraint equation set is solved to generate the compensation coefficients of the corresponding color channels in each original image. The absolute value of the compensation coefficient is less than the preset compensation threshold.

[0122] In this embodiment, the solving mode can be SVD (Singular Value Decomposition), least square method, QR (QR decomposition) decomposition method, etc. The function value G is solved by any of the above modes f = 0 approximation, thereby obtaining the compensation coefficients of the corresponding color channels in each original image.

[0123] Meanwhile, in order to avoid over-compensation leading to image color distortion, the absolute value of the compensation coefficient can be less than or equal to a preset compensation threshold. If the absolute value of the solved compensation coefficient is greater than or equal to the preset compensation threshold, the absolute value of the solved compensation coefficient can be less than or equal to the preset compensation threshold. For example, if the preset compensation threshold is 1.4 and the absolute value of the solved compensation coefficient is 1.5, the solved compensation coefficient can be equal to 1.4 or 1.39, or can be equal to other values less than 1.4. The preset compensation threshold can be set according to the actual camera sensor type, the size of the panorama effect, etc. For example, it can be set to 0.65 to 1.45.

[0124] Optionally, in this embodiment, since there are scenes with too large color difference mutations in actual applications, corresponding processing can be performed when the function value exceeds the preset color difference threshold to solve the problem of too large color difference mutations. S205 can be specifically as follows:

[0125] If the absolute value of the function value of the target constraint equation set is greater than the preset color difference threshold, the compensation coefficient of the corresponding color channel in the largest original image is set to a preset value. The largest original image is the original image to which the target region with the largest average pixel value belongs.

[0126] The target constraint equation set is solved to generate the compensation coefficients of the corresponding color channels in other original images. The other original images are the original images other than the largest original image.

[0127] If the absolute value of the function value of the target constraint equation set is less than or equal to the preset color difference threshold, the target constraint equation set is solved to generate the compensation coefficients of the corresponding color channels in each original image.

[0128] In this embodiment, the preset color difference threshold can be set according to the actual camera sensor type, the size of the panorama effect, etc. For example, it can be set to 0.5. The preset color difference threshold can be set correspondingly according to different color channels, which can be set to the same threshold or different thresholds. If the function value obtained by solving is greater than the preset color difference threshold, for example, the function value corresponding to a certain color channel is greater than the preset color difference threshold, it indicates that the color difference between the original images in this color channel is relatively large. At this time, it can be assumed that one of the original images is the true color of the ambient light, and the solution of other original images on this basis is directly solved. In this embodiment, the original image with the maximum average pixel value in this color channel, that is, the maximum original image, is used as the reference. Since the maximum original image is the true color of the ambient light by default, it does not need to be compensated, and therefore the preset value corresponding to the maximum original image can be set to 1.

[0129] In step S206, the original images are compensated and spliced and converted according to the compensation coefficient to generate a target bird's eye view.

[0130] In this embodiment, the implementation of S206 is similar to that of the previous embodiment S103, and will not be described here.

[0131] In order to further reduce the color difference of the joint area in the bird's eye view, in this embodiment, as shown in Figure 5 , S206 can be further refined.

[0132] In this embodiment, as shown in Figure 6 , the splicing area in the target bird's eye view is an area formed by the new boundary line and the boundary line of the target bird's eye view after the joint line in the target bird's eye view is rotated by a preset center angle to both sides, that is, the areas of A, B, C and D in the figure.

[0133] The new boundary line is the joint line after rotating by a preset center angle. The joint line can be set by the user in advance, for example, the diagonal line position of the bird's eye view can be set as the joint line, or it can be set inside the diagonal line, so that the image content corresponding to the fisheye camera in the left and right directions of the vehicle occupies a larger proportion, that is, the content displayed in the left and right directions of the vehicle occupies a larger proportion in the bird's eye view. The preset center angle can be set by the user in advance, for example, it can be set to 20°, 30°, etc., but generally less than 45°. The center can be the starting position of the joint line close to the vehicle body.

[0134] The image splicing method provided in this embodiment includes the following steps.

[0135] S2061, compensating the target area according to the first included angle, the preset center angle and the compensation coefficient to generate a compensated target area. The first included angle is the angle between the connecting line between the pixels in the splicing area and the center and the new boundary line.

[0136] As shown in Figure 6 , θ in the figure represents the first included angle of the black point (pixel) in the figure, and a in the figure is the central angle of the target area A, which is twice the preset central angle. The intersection line exemplarily shows the in the lower left area. Meanwhile, the first included angle can also be the included angle of the black point (pixel) in the fisheye 2 direction, that is, the included angle between the pixel in the figure and another newly added boundary line, which is a-θ in size.

[0137] Optionally, in the embodiment, 2061:

[0138] The stitching weight of each pixel in the target area is determined according to the first included angle and the preset central angle.

[0139] For each pixel in the target area, the stitching weight, the compensation coefficient and the pixel value of the corresponding pixel in the target area are weighted to generate the pixel value of the corresponding pixel in the compensated target area.

[0140] The compensated target area is generated based on the pixel values of the pixels in the compensated target area.

[0141] In the embodiment, the target area exists in each original image. When generating the bird's eye view, the corresponding target areas between adjacent original images are used to generate the intersection area in the bird's eye view. As shown in Figure 6 , the image area of the fisheye 1 is generated by the original image of the fisheye 1. Similarly, the image areas of the fisheye 2 to the fisheye 4 are also generated by the corresponding original images. There is an intersection area B between the image area of the fisheye 1 and the image area of the fisheye 2, which is generated by the stitching and conversion of the target area 1 of the corresponding original image of the fisheye 1 and the target area 2 of the corresponding original image of the fisheye 2.

[0142] The target area 1 and the target area 2 have corresponding stitching weights, which can be determined by the first included angle and the preset central angle. The pixel values of the pixels in the compensated target area are generated by the weighted processing of the pixel values of the pixels in the target area based on the stitching weight and the compensation coefficient of the target area 1 and the target area 2. The pixel values of the pixels in the compensated target area can be calculated based on the following algorithm:

[0143]

[0144] Wherein, θ is the first included angle, a is the double central angle, rgb1 is the pixel value of the first target area, f1 is the compensation coefficient of the first target area, rgb2 is the pixel value of the second target area, and f2 is the compensation coefficient of the second target area.

[0145] Optionally, in the embodiment, the newly added boundary line has a corresponding relationship with the original image.

[0146] The process of determining the stitching weight of each pixel in the target region according to the first included angle and the preset central angle of the circle can be specifically as follows:

[0147] The second included angle is determined from the first included angle according to the corresponding relationship. The newly added boundary line of the second included angle corresponds to the original image to which the target region belongs.

[0148] The stitching weight corresponding to each pixel in the target region is determined according to the quotient between the second included angle and twice the preset central angle of the circle.

[0149] In this embodiment, each second included angle corresponds to a newly added boundary line. As shown in FIG. 2B, the newly added boundary line is two lines in the handover region B, and one of them is represented in the figure. The first included angle is the included angle between the line formed between the pixel and the center of the circle and the newly added boundary line, and there are two included angles. The second included angle is the included angle corresponding to the original image in the first included angle. Figure 6

[0150] For example, the second included angle of fisheye 1 corresponding to the original image is θ in the figure, and the second included angle of fisheye 2 corresponding to the original image is α-θ in the figure.

[0151] According to the quotient between the second included angle and twice the preset central angle of the circle, the corresponding weight can be calculated. For example, the stitching weight of fisheye 1 corresponding to the original image is the ratio between θ and α, and the stitching weight of fisheye 2 corresponding to the original image is 1-the ratio.

[0152] S2062, compensating the remaining region in each original image according to the compensation coefficient to generate a compensated remaining region. The remaining region is the region in the original image other than the target region.

[0153] In this embodiment, the remaining region does not need to be overlapped and stitched, and can be directly compensated by multiplying the pixel value by the compensation coefficient.

[0154] S2063, stitching and converting the compensated target region and the compensated remaining region to generate a target bird's eye view.

[0155] The stitching and conversion processing can be to first perform corresponding format conversion on the compensated target region and the compensated remaining region to generate a corresponding overhead view, and then perform corresponding stitching processing on the overhead view to generate the final target bird's eye view.

[0156] ​In this embodiment, under poor lighting conditions or motion scenes, the obvious stitching seam and color difference of the panoramic image will cause visual interference to the driver and affect the user experience by using the traditional stitching method. The image stitching method of this embodiment can automatically balance the color and brightness of multiple original images, improve the visibility and accuracy of the panoramic image. At the same time, by using different compensation schemes for the stitching area and other areas, the color difference between the stitching area and the adjacent other areas can be reduced accordingly, thereby effectively alleviating the "stitching seam" phenomenon in the stitching area of the bird's eye view and improving the user experience. For motion scenes and light mutation scenes, corresponding threshold values are defined to avoid overcompensation leading to color distortion and color mutation. For scenes with too large color difference, the scheme content of directly compensating other images according to the maximum original image as the reference is also added, which improves the driving experience of the driver in the whole scene.

[0157] Figure 7 The structure diagram of the image stitching device provided in this application is shown in Figure 7 In this embodiment, the image stitching device 300 can be arranged in an electronic device, and the image stitching device 300 comprises:

[0158] The first determination module 301 is configured to determine the average pixel value of the target area in each of the multiple original images. The target area is the area in the original image corresponding to the stitching area in the finally generated target bird's eye view. The stitching area is the entire overlapping area or part of the overlapping area between each overhead image in the target bird's eye view.

[0159] The second determination module 302 is configured to determine the compensation coefficient corresponding to each original image according to the average pixel value of each target area and a preset constraint equation group.

[0160] The generation module 303 is configured to compensate and stitch each original image according to the compensation coefficient to generate a target bird's eye view.

[0161] The image stitching device provided in this embodiment can implement the technical solutions of the method embodiment shown in Figure 3 The implementation principle and technical effects are similar to those of the method embodiment shown in Figure 3 Therefore, they will not be described here.

[0162] The image stitching device provided in this application further refines the image stitching device provided in the previous embodiment, and the image stitching device 300 comprises:

[0163] Optionally, in this embodiment, the first determination module 301 is specifically configured to:

[0164] The pixels of each target region are converted to the RGB color space. The pixels of the target region are determined based on a preset mapping table and the pixels of the stitching region in the target bird's eye view. The average pixel value corresponding to the RGB color channel in the target region is calculated.

[0165] Optionally, in the embodiment, the second determining module 302 is specifically configured to:

[0166] For each target region, the following processing is performed:

[0167] The average pixel value corresponding to each color channel is input into the corresponding preset constraint equation set to construct a target constraint equation set. The target constraint equation set is solved to generate the compensation coefficient of the corresponding color channel in each original image. The absolute value of the compensation coefficient is less than a preset compensation threshold.

[0168] Optionally, in the embodiment, when the second determining module 302 solves the target constraint equation set to generate the compensation coefficient of the corresponding color channel in each original image, the second determining module 302 is specifically configured to:

[0169] If the absolute value of the function value of the target constraint equation set is greater than a preset color difference threshold, the compensation coefficient of the corresponding color channel in the largest original image is set to a preset value. The largest original image is the original image to which the target region with the largest average pixel value belongs. The compensation coefficient of the corresponding color channel in other original images is generated by solving the target constraint equation set. The other original images are the original images other than the largest original image. If the absolute value of the function value of the target constraint equation set is less than or equal to the preset color difference threshold, the compensation coefficient of the corresponding color channel in each original image is generated by solving the target constraint equation set.

[0170] Optionally, in the embodiment, the stitching region in the target bird's eye view is a region formed by a new boundary line and a boundary line of the target bird's eye view after the intersection line in the target bird's eye view is rotated by a preset center angle based on the center. The new boundary line is the intersection line rotated by the preset center angle.

[0171] The generating module 303 is specifically configured to:

[0172] The target region is compensated according to the first included angle, the preset center angle, and the compensation coefficient to generate a compensated target region. The first included angle is the angle between the connecting line formed by each pixel in the stitching region and the center and the new boundary line. The remaining region in each original image is compensated according to the compensation coefficient to generate a compensated remaining region. The remaining region is the region in the original image other than the target region. The compensated target region and the compensated remaining region are stitched and converted to generate the target bird's eye view.

[0173] Optionally, in this embodiment, the generating module 303, when generating the compensated target region according to the first included angle, the preset central angle of a circle, and the compensation coefficient, is specifically configured to:

[0174] determine the stitching weight of each pixel in the target region according to the first included angle and the preset central angle of a circle; and perform weighted processing on the stitching weight, the compensation coefficient, and the pixel value of the corresponding pixel in the target region, to generate the pixel value of the corresponding pixel in the compensated target region; and generate the compensated target region based on the pixel values of the pixels in the compensated target region.

[0175] Optionally, in this embodiment, the newly added boundary line has a corresponding relationship with the original image.

[0176] The generating module 303, when determining the stitching weight of each pixel in the target region according to the first included angle and the preset central angle of a circle, is specifically configured to:

[0177] determine the second included angle from the first included angle according to the corresponding relationship; the newly added boundary line of the second included angle corresponds to the original image to which the target region belongs; and determine the stitching weight corresponding to each pixel in the target region according to the quotient between the second included angle and twice the preset central angle of a circle.

[0178] Optionally, in this embodiment, the image stitching device 300 further includes:

[0179] a mapping module configured to obtain the calibration parameter corresponding to the historical original image and the historical bird's eye view; inversely calculate the target region of the historical original image based on the calibration parameter and the stitching region of the historical bird's eye view; and store the pixel mapping relationship between the stitching region and the target region in a preset mapping table.

[0180] The image stitching device provided in this embodiment can perform Figures 3-6 the technical solutions of the method embodiments shown in Figures 3-6 are similar to those of the method embodiments shown in, and will not be described here one by one.

[0181] According to embodiments of the present application, the present application also provides an electronic device, a computer-readable storage medium, and a computer program product.

[0182] As shown in Figure 8 , Figure 8 is a structural schematic diagram of an electronic device provided by the present application. The electronic device is intended for various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, vehicle-mounted terminals, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0183] As shown in Figure 8 The electronic device includes a processor 401 and a memory 402. The various components are connected to each other via different buses, and can be mounted on a common main board or otherwise mounted as needed. The processor can process instructions executed within the electronic device.

[0184] The memory 402 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by at least one processor, so that the at least one processor executes the image stitching method provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute the image stitching method provided by the present application.

[0185] The memory 402, as a kind of non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the image stitching method in the embodiments of the present application (for example, the first determination module 301, the second determination module 302 and the generation module 303 shown in the figure). The processor 401 executes various functional applications and data processing of the electronic device by running the non-transitory software programs, instructions and modules stored in the memory 402, that is, implements the image stitching method in the above method embodiments. Figure 7

[0186] Meanwhile, the present embodiment also provides a computer product, when the instructions in the computer product are executed by the processor of the electronic device, the electronic device can execute the image stitching method of the above-mentioned embodiments.

[0187] Those skilled in the art will readily conceive other implementations of the embodiments of the present application upon considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptive changes of the embodiments of the present application that follow the general principles of the embodiments of the present application and include common knowledge or conventional technical means in the technical field of the present application which are not disclosed by the embodiments of the present application.

[0188] It should be understood that the embodiments of the present application are not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the embodiments of the present application is only limited by the appended claims.​

Claims

1. An image stitching method, characterized by, The method comprises the following steps: respectively determining average pixel values of target regions in multiple original images; the target regions are regions in the original images corresponding to splicing regions in a final generated target bird's eye view; the splicing regions are all or part of overlapping regions between each overhead image in the target bird's eye view; determining compensation coefficients corresponding to each original image according to the average pixel values of each target region and a preset constraint equation set; performing compensation processing and splicing conversion processing on each original image according to the compensation coefficients to generate the target bird's eye view.

2. The method of claim 1, wherein, The method of respectively determining average pixel values of target regions in multiple original images comprises the following steps: converting pixels of each target region to an RGB color space; the pixels of the target region are determined based on a preset mapping table and pixels of a splicing region in a target bird's eye view; calculating average pixel values corresponding to RGB color channels in the target region.

3. The method of claim 2, wherein, The method of determining compensation coefficients corresponding to each original image according to the average pixel values of each target region and a preset constraint equation set comprises the following steps: for each target region, the following processing is performed: inputting the average pixel values corresponding to each color channel into a corresponding preset constraint equation set to construct a target constraint equation set; solving the target constraint equation set to generate compensation coefficients in a corresponding color channel in each original image; the absolute value of the compensation coefficient is less than a preset compensation threshold.

4. The method of claim 3, wherein, The method of solving the target constraint equation set to generate compensation coefficients in a corresponding color channel in each original image comprises the following steps: if the absolute value of a function value of the target constraint equation set is greater than a preset color difference threshold, setting the compensation coefficient in the corresponding color channel in the largest original image to a preset value; the largest original image is an original image to which a target region with the largest average pixel value belongs; solving the target constraint equation set to generate compensation coefficients in the corresponding color channel in other original images; the other original images are original images other than the largest original image; if the absolute value of the function value of the target constraint equation set is less than or equal to the preset color difference threshold, solving the target constraint equation set to generate compensation coefficients in the corresponding color channel in each original image.

5. The method of claim 1, wherein, The splicing region in the target bird's eye view is a region formed by a new boundary line and a boundary line of the target bird's eye view after an intersection line in the target bird's eye view is rotated by a preset center angle based on a center of the circle to both sides; The method of generating the target bird's eye view by performing compensation processing and splicing conversion processing on each original image according to the compensation coefficients comprises the following steps: performing compensation processing on the target region according to a first included angle, the preset center angle and the compensation coefficients to generate a compensated target region; the first included angle is an angle between a connecting line formed by each pixel in the splicing region and the center of the circle and the new boundary line; performing compensation processing on a remaining region in each original image according to the compensation coefficients to generate a compensated remaining region; the remaining region is a region in the original image other than the target region; performing splicing conversion processing on the compensated target region and the compensated remaining region to generate the target bird's eye view.

6. The method of claim 5, wherein, The compensation processing on the target region according to the first included angle, the preset central angle of the circle and the compensation coefficient to generate a compensated target region comprises: determining a stitching weight of each pixel in the target region according to the first included angle and the preset central angle of the circle; for each pixel in the target region, performing weighted processing on the stitching weight, the compensation coefficient and the pixel value of the corresponding pixel in the target region to generate a pixel value of the corresponding pixel in the compensated target region; and generating the compensated target region based on the pixel values of the pixels in the compensated target region.

7. The method according to any one of claims 2 to 6, characterized in that, Before the step of determining the average pixel value of the target region in each of the plurality of original images, the method further comprises: obtaining calibration parameters corresponding to historical original images and historical bird's eye view images; based on the calibration parameters, inversely calculating the target region of the historical original images from the stitching region of the historical bird's eye view images; storing a pixel mapping relationship between the stitching region and the target region in a preset mapping table.

8. An electronic device, comprising: comprises: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the image stitching method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the image stitching method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the image stitching method according to any one of claims 1 to 7.

Images