Image processing method and device, surround view system and vehicle

Abstract

The invention relates to an image processing method and device, a surround view system and a vehicle, and the method comprises the steps: obtaining a plurality of to-be-processed images; determining a fusion region of each to-be-processed image, and detecting region pixel points in the fusion region; on the basis of the regional pixel points, calculating a regional equilibrium coefficient in each side fusion region in the to-be-processed image, and calculating a transition equilibrium coefficient of each to-be-processed image according to the regional equilibrium coefficient and the position information of the pixel points in the to-be-processed image; and splicing the to-be-processed image based on the transition equilibrium coefficient, and generating a target panoramic image. According to the invention, the problem of low quality of the panoramic image generated by splicing is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, device, surround view system and vehicle. Background Art

[0002] With the development of science and technology, various types of vehicles are gradually integrating surround view into more models. In-vehicle surround view is a common assisted driving technology. Cameras are installed at the front, rear and left and right rearview mirrors of the vehicle and the target panoramic image is obtained through image processing, which can provide the driver with a rich perspective to observe the surrounding environment, thereby improving driving safety. The panoramic images spliced ​​by the existing surround view system have obvious splicing marks and obvious dividing lines visible to the human eye, which makes the overall display effect of the image inconsistent, seriously affecting the visual effect of the panoramic image and also affecting the driver's judgment of the road conditions.

[0003] Currently, no effective solution has been proposed for the problem of low quality of panoramic images generated by stitching in related technologies. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, surround view system and vehicle to at least solve the problem of low quality of panoramic images generated by stitching in the related art.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Acquire multiple images to be processed;

[0007] Determining a fusion region of each of the images to be processed, and detecting regional pixel points within the fusion region;

[0008] Based on the regional pixels, a regional equalization coefficient of each of the images to be processed in the fusion region is calculated, and according to the regional equalization coefficient and the position information of the pixels in the images to be processed, a transition equalization coefficient of each of the images to be processed is calculated;

[0009] Based on the transition equalization coefficient, the images to be processed are spliced ​​to generate a target panoramic image.

[0010] In some embodiments, the calculating, based on the regional pixels, the regional equalization coefficient of each of the images to be processed within the fusion region includes:

[0011] For the fusion area, determining corresponding adjacent images from each of the images to be processed; the adjacent images include a first image and a second image;

[0012] Based on the regional pixels, calculating a first average value of the first image in the fusion region, and based on the regional pixels, calculating a second average value of the second image in the fusion region;

[0013] Calculating a total pixel value according to the first average value and the second average value;

[0014] The regional equalization coefficient corresponding to the first image is calculated according to the first average value and the total pixel value, and the regional equalization coefficient corresponding to the second image is calculated according to the second average value and the total pixel value.

[0015] In some embodiments, the calculating, based on the regional pixels, a first average value of the first image within the fusion region, and the calculating, based on the regional pixels, a second average value of the second image within the fusion region, comprises:

[0016] Acquire chromaticity difference information between the first image and the second image;

[0017] Based on the chromaticity difference information, assign a first weight value to a first pixel point in the first image corresponding to the pixel point in the region; perform weighted calculation on the pixel value of the first pixel point based on the first weight value to obtain the first average value;

[0018] Based on the chromaticity difference information, a second weight value is assigned to a second pixel point in the second image corresponding to the pixel point in the area; and based on the second weight value, a weighted calculation is performed on the pixel value of the second pixel point to obtain the second average value.

[0019] In some embodiments, the calculating, based on the regional pixels, the regional equalization coefficient of each of the images to be processed within the fusion region includes:

[0020] Based on the regional pixel points, calculating the current frame equalization coefficient corresponding to the image to be processed;

[0021] The regional equalization coefficient is calculated according to the current frame equalization coefficient and the acquired historical frame equalization coefficient.

[0022] In some embodiments, the calculating, based on the regional pixels, the regional equalization coefficient of each of the images to be processed within the fusion region includes:

[0023] Converting the image to be processed into a target color space where the brightness channel and the chrominance channel are separated;

[0024] Calculate the regional equalization coefficient of the brightness channel and / or the chrominance channel of the image to be processed in the target color space.

[0025] In some embodiments, the method further comprises:

[0026] Detecting color difference information of the image to be processed;

[0027] When it is detected that the color difference information is greater than a preset color difference threshold, a regional equalization coefficient of the image to be processed in the luminance channel and a regional equalization coefficient of the image to be processed in the chrominance channel are calculated respectively.

[0028] In some embodiments, the fusion area in each of the images to be processed includes a first side area and a second side area; and calculating the transition equalization coefficient of each of the images to be processed according to the regional equalization coefficient and the position information of the pixel points in the images to be processed includes:

[0029] Calculating a coefficient comparison result between a regional equalization coefficient of the first side area and a regional equalization coefficient of the second side area;

[0030] Based on the coefficient comparison result, a corresponding transition equalization coefficient is allocated to each pixel point according to the position information of each pixel point in the image to be processed.

[0031] In some embodiments, the step of allocating a corresponding transition equalization coefficient to each pixel point based on the coefficient comparison result and according to the position information of each pixel point in the image to be processed includes:

[0032] Based on the position information of each pixel point, determine a first side pixel point and a second side pixel point in the image to be processed, and calculate a position comparison result between the first side pixel point and the second side pixel point;

[0033] Calculating a unit allocation coefficient according to the coefficient comparison result and the position comparison result;

[0034] Taking the regional equalization coefficient of the first side area as the starting point, the unit allocation coefficient is evenly and incrementally allocated to each pixel point according to the position difference between each pixel point and the first side pixel point, and the transition equalization coefficient is obtained.

[0035] In some embodiments, stitching the images to be processed based on the transition equalization coefficient and generating a target panoramic image includes:

[0036] Get the preset grid texture image;

[0037] Based on the position information of each pixel in the image to be processed, storing the transition equalization coefficient in the grid texture image;

[0038] The grid texture image is used to calculate the equalization adjustment parameters for the image to be processed, and the image to be processed is spliced ​​based on the equalization adjustment parameters to generate the target panoramic image.

[0039] In some embodiments, determining the fusion area of ​​each of the images to be processed includes:

[0040] A virtual panoramic image is determined based on each of the images to be processed, and the fusion area in the virtual panoramic image is calculated.

[0041] In some embodiments, detecting regional pixels within the fusion area includes:

[0042] Based on a preset shrinkage coefficient, shrinking the virtual panoramic image to obtain a pixel sampling area;

[0043] Detect pixel points in the pixel sampling area that are located in the fusion area to obtain the regional pixel points.

[0044] In some embodiments, detecting the pixel points in the pixel sampling area that are located in the fusion area to obtain the regional pixel points includes:

[0045] Detecting an initial pixel point in the pixel sampling area that is located within the fusion area;

[0046] Based on a preset sampling step size, the initial pixel points are sampled to obtain the regional pixel points.

[0047] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0048] An acquisition module, used for acquiring a plurality of images to be processed;

[0049] A fusion region sampling module, used to determine the fusion region of each of the images to be processed, and detect regional pixel points within the fusion region;

[0050] An equalization coefficient calculation module, used to calculate the regional equalization coefficient of each of the images to be processed in the fusion area based on the regional pixel points, and calculate the transition equalization coefficient of each of the images to be processed according to the regional equalization coefficient and the position information of the pixel points in the image to be processed;

[0051] A generation module is used to splice the images to be processed based on the transition equalization coefficient and generate a target panoramic image.

[0052] In a third aspect, an embodiment of the present application provides a surround view system, including: an image acquisition device and a controller;

[0053] The image acquisition device is used to acquire a plurality of images to be processed;

[0054] The controller is connected to the image acquisition device, and is used to acquire each of the images to be processed and execute the image processing method as described in the first aspect above.

[0055] In a fourth aspect, an embodiment of the present application provides a vehicle, comprising a surround view system as described in the third aspect above.

[0056] Compared with the related art, the image processing method, device, surround view system and vehicle provided in the embodiments of the present application obtain multiple images to be processed; determine the fusion area of ​​each image to be processed, and detect the regional pixel points in the fusion area; based on the regional pixel points, calculate the regional equalization coefficient in the fusion area on each side of the image to be processed, and calculate the transition equalization coefficient of each image to be processed according to the regional equalization coefficient and the position information of the pixel points in the image to be processed; based on the transition equalization coefficient, splice the images to be processed and generate a target panoramic image. Based on this, in the process of transitioning from one side of each image to be processed to the other side, the equalization coefficient of each pixel point will also gradually change, so that the brightness value of the image can be evenly adjusted. When the fisheye cameras around the vehicle are in a complex scene with large lighting differences, the display effects such as brightness between two adjacent splicings will transition evenly, and the overall splicing effect will be significantly improved, which effectively solves the problem of low quality of the panoramic image generated by splicing, and makes the image splicing look more natural.

[0057] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0059] Figure 1 is a hardware structure block diagram of a terminal of an image processing method according to an embodiment of the present application;

[0060] Figure 2 is a flowchart of an image processing method according to an embodiment of the present application;

[0061] Figure 3 is a flowchart of another image processing method according to an embodiment of the present application;

[0062] Figure 4 is a schematic diagram of the overall process of an image processing method according to an embodiment of the present application;

[0063] Figure 5 It is a structural block diagram of an image processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0065] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0066] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0067] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 is a hardware structure block diagram of a terminal of an image processing method according to an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0068] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0069] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0070] This embodiment provides an image processing method. Figure 2 is a flowchart of an image processing method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0071] Step S210, obtaining a plurality of images to be processed.

[0072] Among them, each image to be processed can be collected by a multi-channel image acquisition device, and each image to be processed is usually facing a collection environment in a different direction. Taking the surround view system applied to the vehicle as an example, the surround view system includes a multi-channel surround view image acquisition device deployed around the vehicle, and the multi-channel surround view image acquisition device is generally composed of four fisheye cameras. Then the four fisheye cameras respectively collect the images of the surrounding environment of each vehicle, and in actual application, the fisheye images collected at the same time are spliced ​​into a panoramic image, so as to provide the road environment around the vehicle to the driver comprehensively and accurately. It should be understood that due to the differences in the arrangement of fisheye cameras caused by different vehicle body shapes, the complex lighting when the vehicle is driving, and the influence of the algorithms of each camera, the panoramic image splicing effect in the prior art usually has brightness and color differences, so that there is an obvious "dividing line" in the panoramic image, which seriously affects the visual effect of the panoramic image; based on this, this embodiment performs equalization processing on the currently collected image to be processed through subsequent steps.

[0073] It should also be noted that the above-mentioned multi-channel image acquisition equipment for acquiring each image to be processed can be used not only in the surround view system, but also in other application scenarios that require splicing various images to generate panoramic images, such as traffic intersections, etc., which will not be repeated here.

[0074] Step S220, determining the fusion area of ​​each image to be processed, and detecting regional pixel points in the fusion area.

[0075] The fusion area specifically refers to the overlapping area between two adjacent images in each image to be processed. The fusion area can be calculated based on algorithms such as image features, geometric transformation or pixel difference. Usually, there is a fusion area on each side of the image to be processed; for example, there is a fusion area on the left side of the i-th image, and the images corresponding to the left fusion area are the i-1-th image and the i-th image, and there is another fusion area on the right side of the i-th image, and the images corresponding to the right fusion area are the i-th image and the i+1-th image; i is a positive integer greater than 1.

[0076] In this step, for each detected fusion area, the pixel points in each image to be processed can be traversed to detect whether they are located in the fusion area. Alternatively, a virtual panoramic image at the time of stitching can be obtained first, and all pixel point coordinates in the virtual panoramic image can be traversed to detect whether the currently traversed pixel point coordinates are located in the fusion area, so as to obtain the regional pixel points in each fusion area.

[0077] Step S230, based on the regional pixels, calculate the regional equalization coefficient of each image to be processed in the fusion area, and calculate the transition equalization coefficient of each image to be processed according to the regional equalization coefficient and the position information of the pixels in the image to be processed.

[0078] The regional equalization coefficient is a quantitative indicator for evaluating the degree of balance of the characteristics (such as brightness, contrast, etc.) of each pixel point of the image to be processed in a specific fusion area. The calculation process of the regional equalization coefficient is described below. First, the fusion area of ​​each image to be processed is determined according to the above steps; for each image to be processed, the pixel values ​​in the fusion area on each side are extracted. In this way, for each fusion area, the pixel values ​​of the corresponding two adjacent images to be processed can be determined, and then the statistics of the pixel points of the two adjacent images, such as the mean, variance, standard deviation, etc., can be determined. These statistics will be used to evaluate the brightness, contrast and other characteristics of each image in the fusion area. Then, the statistics of the above two adjacent images are combined to calculate the regional equalization coefficient of each image in the fusion area. This coefficient can reflect the overall performance of the image in the fusion area, such as whether the brightness is uniform, whether the contrast is moderate, etc. The specific calculation method can be set according to actual needs. For example, the ratio of the weighted mean to the standard deviation can be used as a measure of the equalization coefficient. Through the above calculation of the regional equalization coefficient of the fusion area, the brightness of the two adjacent images can be taken into account.

[0079] Next, based on the regional equalization coefficients of the fusion areas on both sides of each image to be processed, the transition equalization coefficients in each image are calculated. The transition equalization coefficient is a quantitative index that evaluates the comprehensive performance of the transition of the image to be processed in the fusion areas on both sides (such as the brightness and contrast of the transition area) after further considering the position information of the pixels in the image. Among them, for each pixel in the image to be processed, its position information is obtained, including the coordinates in the image and the position relative to the fusion area (such as the distance from the boundary of the fusion area). According to the position information of the pixel, a weight function is set. This function should be able to reflect the importance of the pixel in the transition area. For example, the closer the pixel is to the boundary of the fusion area, the greater its weight should be. Then, the transition equalization coefficient of each image to be processed is calculated by combining the regional equalization coefficient and the weight coefficient. The transition equalization coefficient can be regarded as the comprehensive performance of the image in the transition area, which takes into account both the balance of the image in the fusion area and the position information of the pixel. The specific calculation method can be to multiply the regional equalization coefficient by the weight of the corresponding pixel, and perform weighted summation on all pixels.

[0080] It can be seen that the above transition equalization coefficient shows a preset change trend along a preset direction (usually a horizontal direction) in the corresponding image to be processed. The preset direction and the preset change area are determined according to the specific calculation method of the transition equalization coefficient. For example, considering that each image to be processed is usually collected by looking around in the horizontal direction, in this embodiment, the horizontal distance between each pixel point in each image to be processed and the boundary position of the fusion area can be calculated, and an arithmetically increasing equalization coefficient is assigned to each pixel point according to the distance by weighted calculation, so that the transition equalization coefficient in the image to be processed shows a uniformly increasing change trend from one side to the other as a whole. Or, an arithmetically increasing equalization coefficient is assigned to each pixel point according to the calculated horizontal distance, so that the transition equalization coefficient in the image to be processed shows a uniformly decreasing change trend from one side to the other as a whole.

[0081] In addition, it should be noted that for each channel of images to be processed, due to the influence of the algorithm, there may be image parts outside the fusion area during stitching. For example, for the i-th image, the horizontal coordinate of the leftmost pixel in the left fusion area is x l , the horizontal coordinate of the rightmost pixel in the right fusion area is x R , and there are still horizontal coordinates less than x in the road image. l Pixels with horizontal coordinates greater than x R These pixels are the pixels outside the fusion area. In order to improve the efficiency and accuracy of image equalization processing, for these pixels, the corresponding equalization coefficients can be uniformly set to 1, that is, no equalization compensation processing is required for this part.

[0082] Through the above steps, the regional equalization coefficient can be calculated based on the regional pixels in the fusion area, and the transition equalization coefficient can be calculated based on the position information of the pixels. These coefficients can be used in subsequent image stitching and fusion processing to achieve a smoother and more natural transition effect.

[0083] Step S240: splicing the images to be processed based on the transition equalization coefficients to generate a target panoramic image.

[0084] Specifically, according to the transition equalization coefficient in each image to be processed and the original pixel value of the image to be processed, the compensation difference of each pixel point in the image to be processed can be calculated. In the process of splicing the images to be processed, the original pixel value is balanced and compensated according to the compensation difference of each pixel point, and finally a target panoramic image of the balanced processing is generated.

[0085] In the above-mentioned image processing method, by calculating the equalization coefficient of the fusion area on both sides of each image to be processed, and calculating the transition equalization coefficient of each image to be processed according to the regional equalization coefficient and the position information of the pixel points in the image to be processed, the equalization coefficient of each pixel point will gradually change during the transition from one side of each image to be processed to the other side, so that the brightness value of the image can be evenly adjusted. When the fisheye cameras around the vehicle are in a complex scene with large lighting differences, the display effects such as brightness between two adjacent stitching will transition evenly, and the overall stitching effect will be significantly improved, which effectively solves the problem of low quality of the panoramic image generated by stitching, and makes the image stitching look more natural.

[0086] In some of the embodiments, the above-mentioned determination of the fusion area of ​​each image to be processed may also include the following steps: determining a virtual panoramic image based on each image to be processed, and calculating the fusion area in the virtual panoramic image. Specifically, in the stitching process, after preprocessing and coordinate change of each image to be processed, a virtual panoramic pixel sampling area is obtained. This area may not directly correspond to an actual image file, but is composed of transformed pixels. We can traverse each pixel point of this virtual area and determine whether these pixels are in the fusion area of ​​the two image acquisition devices according to the selected fusion area determination method.

[0087] Through the above-mentioned embodiment, the panoramic image stitching process and the equalization coefficient calculation process are synchronized and executed separately in different threads, so that the image equalization processing is completed during the stitching process, which effectively improves the image processing efficiency.

[0088] In some embodiments, the above-mentioned detection of regional pixels within the fusion area may further include the following steps:

[0089] Based on a preset shrinkage coefficient, the virtual panoramic image is shrunk to obtain a pixel sampling area; and the pixel points in the pixel sampling area that are located in the fusion area are detected to obtain regional pixel points.

[0090] The above shrinkage coefficient can be set in advance by the staff according to the actual situation. Specifically, the shrinkage coefficient S in the horizontal direction can be selected. x and the vertical shrinkage coefficient S y , where S x ,S y ∈[0,1]. The virtual panoramic image, namely the Bird Eye View (BEV) interface, is horizontally shrunk by the coefficient S x Shrinking inwards along the horizontal direction, and according to the vertical shrinkage coefficient S yThe area after shrinking inward along the vertical direction is used as the pixel sampling area. In this embodiment, the pixels in the pixel sampling area are traversed, and it is determined whether each pixel is located in the fusion area.

[0091] Through the above embodiment, pixel sampling is only performed on the shrunk area, which can effectively reduce the impact caused by severe edge distortion of images captured by image acquisition devices such as fisheye cameras, thereby improving the accuracy of image processing.

[0092] In some embodiments, the above-mentioned detection of pixel points in the pixel sampling area located in the fusion area to obtain regional pixel points may also include the following steps:

[0093] Detect the initial pixel points in the pixel sampling area that are located in the fusion area; based on a preset sampling step size, sample the initial pixel points to obtain regional pixel points.

[0094] Specifically, after initially detecting whether each pixel in the pixel sampling area is located in the fusion area, a relatively dense pixel set in the fusion area can be obtained. In order to improve the image processing efficiency, in this embodiment, the pixel set can be further sparsely sampled. That is, a suitable fixed sampling step size is selected, and the pixel points (x bev ,y bev ), and x bev ∈[W bev ·S x ,W bev ·(1-S x )],y bev ∈[H bev ·S y ,H vev ·(1-S y )], if (x bev ,y bev ) is located in the BevMixArea ij , then the pixel coordinates of the point stored on the BEV interface are (x bev,ij ,y bev,ij ), all located in the fusion area BevMixArea ij The number of points is N ij Among them, W bbv Used to indicate the width of the virtual panoramic image, H bev Used to indicate the height of the virtual panoramic image. Through the above embodiment, some pixels are selected for processing by sparse point sampling, thereby significantly reducing the amount of calculation and correspondingly improving the image processing speed.

[0095] In some embodiments, the above-mentioned calculation of the regional equalization coefficient of each image to be processed in the fusion area based on the regional pixels may also include the following steps:

[0096] The image to be processed is converted into a target color space in which the brightness channel and the chrominance channel are separated; and the regional equalization coefficients of the brightness channel and / or the chrominance channel of the image to be processed in the target color space are calculated.

[0097] Common color spaces with separated brightness channels and chrominance channels include YUV, YCbCr, Lab, etc. If the image to be processed is not in such a color space, the color space conversion is performed to convert the image to be processed from the original space image (such as RGB) to the target color space. Taking the YUV color space as an example, the conversion process can be expressed by the following formula:

[0098]

[0099] After the image to be processed is converted to the target color space by the above formula, the equalization coefficients under the Y channel, U channel and / or V channel can be calculated respectively, so that the brightness and / or color difference can be equalized respectively based on the calculated equalization coefficients. Compared with the method of calculating the overall color equalization coefficient in other color spaces, in this embodiment, the image to be processed is converted to the target color space, and the equalization coefficients under each channel are calculated, which can facilitate the equalization of different channels such as brightness and color difference separately, so that the image processing method can be more suitable for different application scenarios.

[0100] Compared with the prior art which is usually limited to adjusting a single Y channel in the YUV color space, it is often difficult to accurately restore color differences under complex lighting conditions, which results in deviations in the overall color of the image and affects the visual quality. Through the above steps, the embodiment of the present application can not only balance the Y channel, but also expand it to the U and V channels, thereby achieving comprehensive optimization of the image color information, and can perform gradual adjustments on all three YUV channels, which can effectively improve the color difference problem caused by uneven brightness under complex lighting conditions.

[0101] In some embodiments, the above image processing method may further include the following steps:

[0102] Detect the color difference information of the image to be processed. According to the characteristics of the image to be processed and the application requirements, select a suitable color difference detection algorithm. The color difference detection algorithm can be based on the distance metric in the color space (such as Euclidean distance, Manhattan distance, etc.), or based on a specific color difference formula. Apply the color difference detection algorithm to the image to be processed to calculate the color difference information of each pixel or area in the image; the color difference information can be expressed as the size or degree of color difference, usually a numerical value or vector.

[0103] Next, according to the application requirements, one or more color difference thresholds are set; the color difference threshold is used to determine whether the color difference in the image is significant and whether further equalization processing is required. The color difference information calculated above is compared with the preset color difference threshold. When it is detected that the color difference information is greater than the preset color difference threshold, it can be considered that the color difference in the image to be processed is relatively obvious. At this time, in addition to calculating the regional equalization coefficient of the image to be processed in the brightness channel to equalize the image brightness, it is also necessary to separately calculate the regional equalization coefficient of the image to be processed in the chroma channel. In the subsequent steps, the transition equalization coefficient under each channel will be calculated separately according to the regional equalization coefficient under the brightness channel and the regional equalization coefficient under the chroma channel, so as to achieve consistent equalization processing of the image brightness and color difference. On the contrary, it can be considered that the color difference in the image is not obvious, and there is no need to equalize the color difference. At this time, it is only necessary to calculate the regional equalization coefficient of the image to be processed in the brightness channel and equalize the image brightness.

[0104] Through the above embodiment, the color difference information of the image to be processed is calculated to determine whether the image still needs to be subjected to color difference equalization processing, thereby further optimizing the equalization processing method for each channel in the target color space and improving the processing efficiency.

[0105] In some embodiments, the above-mentioned calculation of the regional equalization coefficient of each image to be processed in the fusion area based on the regional pixels may also include the following steps:

[0106] For the fusion area, the corresponding adjacent images are determined from each image to be processed; the adjacent images include a first image and a second image; based on the regional pixel points, a first average value of the first image in the fusion area is calculated, and based on the regional pixel points, a second average value of the second image in the fusion area is calculated; according to the first average value and the second average value, a total pixel value is calculated; according to the first average value and the total pixel value, a regional equalization coefficient corresponding to the first image is calculated, and according to the second average value and the total pixel value, a regional equalization coefficient corresponding to the second image is calculated.

[0107] First, from all the images to be processed, it is determined which images are adjacent according to the position of the fusion region. Taking a certain fusion region as an example, there are two corresponding adjacent regions in the fusion region, which are respectively the first image and the second image.

[0108] Extract all pixels in the fusion area from the first image and the second image respectively, calculate the average value of all pixels in the fusion area of ​​the first image, and the left and right average values ​​of the second image in the fusion area. Then, add the first average value and the second average value to get the total pixel value; the total pixel value represents the overall brightness or color level of all pixels in the fusion area.

[0109] Next, the ratio between the first average value and the total pixel value is used to calculate the regional equalization coefficient of the first image in the fusion area; the regional equalization coefficient reflects the relative brightness or color contribution of the first image during the fusion process. Similarly, the ratio between the second average value and the total pixel value is used to calculate the regional equalization coefficient of the second image in the fusion area.

[0110] More specifically, taking the calculation of the regional equalization coefficient under the Y channel as an example, the calculation process of the above regional equalization coefficient can be expressed by the following formula:

[0111]

[0112] in, It is used to represent the first average value of the i-th image to be processed in the Y channel. It is used to represent the second average value of the j-th image to be processed in the Y channel; the i-th image to be processed and the j-th image to be processed are adjacent images. It is used to represent the regional equalization coefficient of the i-th image to be processed in the Y channel. It is used to represent the regional equalization coefficient of the j-th image to be processed under the Y channel; and K Y,i～ij , K Y,j～ij ∈[0,2].

[0113] Through the above embodiments, in the actual image stitching process, the brightness or color of two adjacent images in the fusion area can be adjusted by the regional equalization coefficients calculated for each adjacent image to ensure a more natural and smooth transition, thereby further improving the image processing quality.

[0114] In some embodiments, the above-mentioned calculation of the first average value of the first image in the fusion area based on the regional pixels and the calculation of the second average value of the second image in the fusion area based on the regional pixels may further include the following steps:

[0115] Acquire chromaticity difference information between a first image and a second image; assign a first weight value to a first pixel point corresponding to a regional pixel point in the first image based on the chromaticity difference information; perform weighted calculation on the pixel value of the first pixel point based on the first weight value to obtain a first average value; assign a second weight value to a second pixel point corresponding to the regional pixel point in the second image based on the chromaticity difference information; perform weighted calculation on the pixel value of the second pixel point based on the second weight value to obtain a second average value.

[0116] Specifically, firstly, a first pixel point corresponding to the regional pixel point in the first image and a second pixel point corresponding to the regional pixel point in the second image are determined. For example, the regional pixel point is detected and determined based on the above virtual panoramic image. At this time, the coordinates of the regional pixel point in the fusion area can be expressed as (x bev,ij ,y bev,ij ); According to (x bev,ij ,y bev,ij ), the coordinates of the first pixel point of the i-th image to be processed (x cam,i～ij ,y cam,i～ij ) and the coordinates of the second pixel point (x cam,j～ij ,y cam,j～ij ).

[0117] For the pixels corresponding to the fusion area in the first image and the second image, their differences in the chromaticity space are calculated. This can be achieved by calculating the difference between the two pixels in the chromaticity component. Next, for each pixel corresponding to the fusion area in the first image (referred to as the first pixel), a weight value (referred to as the first weight value) is assigned according to the chromaticity difference information between it and the corresponding pixel in the second image. The smaller the chromaticity difference, the closer the two pixels are in color, so a higher weight can be given. Similarly, for the corresponding pixel in the second image (referred to as the second pixel), a weight value (referred to as the second weight value) is also assigned according to the chromaticity difference information. In the actual image stitching process, these weighted averages can be used to adjust the pixel values ​​of the two images in the fusion area to ensure a more natural and smooth transition. For example, the pixels in the fusion area can be weighted averaged according to these averages to obtain the final stitching result.

[0118] It should also be noted that in order to convert the chromaticity difference into a weight value, a weight function can be defined. This function can be linear, nonlinear, or based on a specific rule. For example, an inverse proportional function can be used so that the smaller the chromaticity difference, the greater the weight; or a threshold function can be used so that when the chromaticity difference is less than a certain threshold, the weight is 1, otherwise it is 0 (or a smaller value).

[0119] Finally, for each first pixel point in the first image, the pixel value is weighted by using the first weight value corresponding to it. Then, all weighted pixel values ​​are averaged to obtain a first average value. Similarly, for each second pixel point in the second image, the pixel value is weighted by using the second weight value corresponding to it. Then, all weighted pixel values ​​are averaged to obtain a second average value.

[0120] More specifically, taking the Y channel as an example, the calculation process of the weighted average of the i-th channel to be processed image and the j-th channel to be processed image is shown in the following formula:

[0121]

[0122] In the above formula, Y x,ycam,i～ij , U x,ycam,i～ij , V x,ycam,i～ij are the values of the Y channel, U channel, and V channel of the i-th fisheye camera at (x cam,i～ij , y cam,i～ij ). is the weighted average of the Y channels of all points (x ij , y bbv,ij , y bev,ij ) in the fusion area BevMixArea is the weighted average of the Y channels of all points (x ij , y bev,ij , y bev,ij ) in the fusion area BevMixArea

[0123] Through the above embodiments, a weighted average method is adopted in the process of solving the equalization coefficient, avoiding the limitations of traditional unweighted average processing. The final equalization coefficient is solved by more carefully considering the contributions of each sampling point of each channel, achieving a more excellent brightness and color difference equalization effect. Compared with the result of unweighted average, the final brightness and color difference balance effect calculated by using the weighted average method is better.

[0124] In some of these embodiments, calculating the regional equalization coefficient of each image to be processed in the fusion area based on the regional pixel points may further include the following steps:

[0125] Based on the regional pixel points, calculate the current frame equalization coefficient corresponding to the image to be processed; according to the current frame equalization coefficient and the obtained historical frame equalization coefficient, calculate the regional equalization coefficient.

[0126] Among them, through the regional equalization coefficient calculation method described in any of the above embodiments, what is calculated is the equalization coefficient of the image to be processed in the current frame. Taking into account that in the process of the image acquisition device acquiring a series of frame images, there may be problems such as brightness jumps in a certain frame; based on this, in this embodiment, in order to ensure the accuracy of image processing, the regional equalization coefficients of N frames of images to be processed acquired at historical moments are also combined for comprehensive calculation; N is a positive integer, and the value range of N can be adjusted according to the application scenario. Based on the current frame equalization coefficient and the historical frame equalization coefficient, an average value or a weighted average calculation can be performed to obtain the final regional equalization coefficient.

[0127] It should be understood that the above-mentioned historical frame equalization coefficients can be calculated for the image to be processed of the historical frame in the same way as the current frame equalization coefficients. These historical frame equalization coefficients may be stored in a data structure, such as an array, a queue or a database. Taking the array as an example, the current frame equalization coefficients of the current frame image to be processed calculated above will be stored in the coefficient storage array of the historical frame equalization coefficients; when the length of the equalization coefficient array of the historical frame exceeds the set length, the oldest equalization coefficient is discarded and the latest equalization coefficient is stored. It should also be noted that the regional equalization coefficients calculated for the current frame image to be processed in combination with the historical frame equalization coefficients will not be stored in the coefficient storage array to ensure that the equalization coefficients of the historical frame will not be affected by the output results of the brightness adjustment algorithm.

[0128] Through the above embodiment, the regional equalization coefficient of the image to be processed is calculated in combination with the historical frame equalization coefficient, thereby avoiding the problem of image processing failure or error caused by sudden brightness changes in a certain frame of image due to environmental factors, and effectively improving the accuracy of image processing.

[0129] In some embodiments, the fusion area in each image to be processed includes a first side area and a second side area; the first side area in the image to be processed may be the fusion area on the right side of the image, and the second side area may be the fusion area on the left side; or vice versa.

[0130] The above-mentioned step of calculating the transition equalization coefficient of each image to be processed according to the regional equalization coefficient and the position information of the pixel points in the image to be processed may also include the following steps:

[0131] Calculate the coefficient comparison result between the regional equalization coefficient of the first side area and the regional equalization coefficient of the second side area; based on the coefficient comparison result, assign a corresponding transition equalization coefficient to each pixel according to the position information of each pixel in the image to be processed.

[0132] The difference between the regional equalization coefficient of the first side area and the regional equalization coefficient of the second side area is compared to determine the difference between the two; this difference reflects the inconsistency of the brightness or contrast of the image to be processed in the fusion area on both sides. Based on the coefficient comparison result obtained by the calculation and the position information of each pixel point in the image to be processed (such as coordinates, distance from the boundary of the fusion area, etc.), a corresponding transition equalization coefficient is allocated to each pixel point. This allocation process can reflect the change trend of the pixel value in the transition area and ensure that the transition equalization coefficient presents a smooth change trend in the image.

[0133] The above-mentioned method of assigning equalization coefficients to each pixel point can be: according to the distance between the pixel point and the boundary of the fusion area, a linear interpolation method is used to calculate the transition equalization coefficient. For example, if the regional equalization coefficient of the first side area is smaller than the regional equalization coefficient of the second side area, the transition equalization coefficient can be gradually increased from the first side area to the second side area. Alternatively, the transition equalization coefficient can be calculated according to the position information of the pixel point based on a specific nonlinear function (such as an exponential function, a logarithmic function, etc.). Alternatively, it can be based on weight allocation; that is, a weight function is set, which assigns a weight to each pixel point according to the position information of the pixel point (such as distance, coordinates, etc.). Then, the regional equalization coefficient is multiplied by the weight of the corresponding pixel point to obtain the transition equalization coefficient.

[0134] It should be noted that when calculating the transition equalization coefficient, for the pixels outside the fusion area, their corresponding equalization coefficients can be uniformly set to 1 (or some other appropriate value), that is, no equalization compensation processing is performed on this part. This can improve the efficiency and accuracy of image processing.

[0135] Through the above embodiments, the transition equalization coefficient of each image to be processed can be calculated based on the regional equalization coefficient and the position information of the pixel points, and the calculated transition equalization coefficient can be applied to the image to be processed to adjust the image's characteristics such as brightness or contrast in the transition area to achieve a smoother and more natural transition effect for the panoramic image.

[0136] In some embodiments, the above-mentioned allocating a corresponding transition equalization coefficient to each pixel point based on the coefficient comparison result according to the position information of each pixel point in the image to be processed may also include the following steps:

[0137] Based on the position information of each pixel point, the first side pixel point and the second side pixel point in the image to be processed are determined, and the position comparison result between the first side pixel point and the second side pixel point is calculated. Wherein, based on the position information of each pixel point, the first side pixel point and the second side pixel point in the image to be processed are determined. This is usually achieved by checking the coordinates of the pixel point or its relative position to the boundary of the fusion area. Generally, the x-coordinate difference between the leftmost pixel sampling point and the rightmost pixel sampling point in the image to be processed is calculated as the above-mentioned position comparison result.

[0138] The unit allocation coefficient is calculated according to the coefficient comparison results and the position comparison results; taking the regional equalization coefficient of the first side area as the starting point, the unit allocation coefficient is evenly and incrementally allocated to each pixel point according to the position difference between each pixel point and the first side pixel point, and the transition equalization coefficient is obtained.

[0139] The above-mentioned unit allocation coefficient is used to determine the increment or decrement of the transition equalization coefficient allocated to each pixel. The size and direction (increasing or decreasing) of the unit allocation coefficient depends on the comparison result between the first average equalization coefficient and the second average equalization coefficient, and the positional relationship between the pixel and the first side pixel. After that, taking the first average equalization coefficient as the starting point, the unit allocation coefficient is uniformly increased or decreased and allocated to each pixel according to the position difference (such as distance, coordinate difference, etc.) between each pixel and the first side pixel. Therefore, each pixel will get a transition equalization coefficient, which reflects its balance in the transition area. Finally, the calculated transition equalization coefficient is applied to the image to be processed to adjust the characteristics of the image such as brightness or contrast in the transition area. This can be achieved by multiplying the transition equalization coefficient with the image pixel value (or other appropriate operations).

[0140] For ease of understanding, the calculation process is described below in conjunction with actual application scenarios. The above transition equalization coefficients can be stored in a grid texture image. For the i-th image, the x coordinates of the leftmost and rightmost pixels in the image can be mapped to the grid texture image to obtain the horizontal coordinates on the grid texture image. That is, the first side pixel point coordinates and the second side pixel point coordinates.

[0141] Taking the Y channel as an example, the transition equalization coefficient The calculation method is shown in the following formula:

[0142]

[0143] in, The pixel coordinates of the equalization coefficients allocated by the current calculation of the image to be processed are mapped to the pixel coordinates under the grid texture image. BevMixArea is the fusion area of ​​the i-th image to be processed ij The application equalization coefficient of the Y channel, that is, the regional equalization coefficient of the first side area; BevMixArea is the fusion area on the other side of the i-th image to be processed ij′ The application equalization coefficient of the Y channel of the second side region is the regional equalization coefficient of the second side region. j and j′ respectively represent two image acquisition devices deployed on both sides of the i-th image acquisition device that acquires the i-th image to be processed, namely, the j-th image acquisition device and the j′-th image acquisition device.

[0144] Through the above embodiment and the above steps, the transition equalization coefficients of each image to be processed can be calculated more accurately, and these coefficients can be applied in subsequent image processing to achieve a smoother and more natural transition effect.

[0145] In some embodiments, the above-mentioned steps of stitching the to-be-processed images based on the transition equalization coefficient and generating the target panoramic image may further include the following steps:

[0146] Get the preset grid texture image. Specifically, create n M×N×Channel grid textures; the value of n is determined by the number of images to be processed. For example, if there are four channels of images to be processed, the value of n is set to 4 accordingly. The values ​​of M and N can be set in advance by the staff based on experience and actual conditions. For example, both M and N can be set to 32, and the grid texture image is composed of 32×32 grids. The number of channels Channel = 3, and the transition equalization coefficients of each channel are stored separately (since the grid texture is an image, the transition equalization coefficient can be mapped from [0,2] to [0,255] in the texture).

[0147] Based on the position information of each pixel in the image to be processed, the transition equalization coefficient is stored in the grid texture image. It should also be noted that in order to accurately store the transition equalization coefficient, the coordinates in the image to be processed need to be mapped to the grid texture image.

[0148] For example, for the i-th image to be processed, the x coordinates of its leftmost and rightmost sampling points are x cam,imaxL 、x cam,imaxR , for x cam,imaxL 、x cam,imaxR The coordinates are normalized to obtain the normalized coordinates Will Horizontal coordinates mapped to an M×N grid texture The calculation method is as follows,

[0149]

[0150] Among them, floor() represents rounding down. Taking the Y channel as an example, traverse each point of the M×N grid texture. When the horizontal coordinate of the M×N grid texture or When the YUV channel equalization coefficient is set to 1, when the horizontal coordinate of the M×N grid texture Then the Y channel is The equilibrium coefficient It can be calculated using the above transition equalization coefficient calculation formula.

[0151] Finally, the grid texture image is used to calculate the equalization adjustment parameters for the image to be processed, and the image to be processed is stitched together based on the equalization adjustment parameters to generate the target panoramic image. Among them, the equalization adjustment parameter is the difference between the transition equalization coefficient stored in the calculated grid texture image and the channel value of the image to be detected. When performing BEV stitching, when sampling the image to be processed through the texture coordinates, the grid texture is sampled using the same texture coordinates. Therefore, for the channel of each sampling point in the image to be processed, the corresponding application equalization coefficient can be found in the grid texture. The detailed adjustment process is as follows:

[0152] Taking the case where the image to be processed needs to be converted to the target color space as an example, the RGB value of the texture of the i-th image to be processed at each texture coordinate is obtained and converted to the YUV color space. The values ​​of the three channels of the M×N grid texture at the same texture coordinates are obtained respectively, that is, the application equalization coefficient of the YUV channel of the M×N grid texture at the same texture coordinates is obtained. The difference ΔY, ΔU, ΔV between the YUV channel value under the application equalization coefficient of the YUV channel at the same texture coordinates and the YUV channel value in the original image is calculated, and ΔY, ΔU, ΔV are converted into the difference ΔR, ΔG, ΔB in the RGB color space. The conversion process is shown in the following formula:

[0153]

[0154] Through the above method, the balanced image RGB value R is obtained dst , G dst , B dst They are;

[0155]

[0156] Through the above-mentioned embodiment, the transition equalization coefficient is stored in the created grid texture to complete the equalization processing of the image to be detected and the panoramic image stitching, so as to facilitate unified processing when using professional graphics program interfaces such as OpenGL; compared to storing the transition equalization coefficient as other storage forms such as matrices, which requires converting matrix data into image data during image processing, this embodiment stores the transition equalization coefficient directly in the grid texture image, which eliminates the need for additional data format conversion, and is conducive to improving the convenience and efficiency of image processing.

[0157] The embodiments of the present application are described and illustrated below in conjunction with specific application scenarios. Taking the application in a surround view system as an example, the surround view system is composed of a plurality of fisheye cameras arranged around a vehicle. Figure 3 is a flowchart of another image processing method according to an embodiment of the present application. Figure 3 As shown, the process includes the following steps:

[0158] Step S301, initializing parameters, determining a fusion area in the BEV sampling area, and sampling the fusion area at fixed step intervals.

[0159] Among them, the width and height of the BEV interface, the relevant parameters of the vehicle are obtained, and the relevant variables are initialized. The horizontal contraction coefficient S is selected x and the vertical shrinkage coefficient S y , the BEV interface is converted into a horizontal contraction coefficient S x Shrinking inwards along the horizontal direction, and according to the vertical shrinkage coefficient S y The area after shrinking in the vertical direction is used as the BEV pixel sampling area. After that, a fusion area determination method is selected to determine the fusion area, and each pixel in the BEV pixel sampling area is traversed to determine whether each pixel is in the fusion area of ​​the two fisheye cameras.

[0160] Select a suitable fixed step size, and traverse the pixels in the BEV pixel sampling area at fixed step intervals. If the pixel is located in the fusion area BevMixArea ij , then store the pixel coordinates of the point on the BEV interface and record the total number of all points in the fusion area.

[0161] Step S302 , based on the coordinates of the pixel points sampled in the fusion area, the coordinates of the sampling points of the two fisheye camera image textures corresponding to the fusion area can be calculated.

[0162] Step S303, calculate the weighted average of the Y channel, U channel, and V channel according to the pixel points of the two fisheye cameras corresponding to the pixel points of the BEV fusion area obtained in step S302, and calculate the equalization coefficients of the Y channel, U channel, and V channel of the fusion area according to the weighted average of the Y channel, U channel, and V channel.

[0163] Step S304, the equalization coefficient calculated by step S303 is the equalization coefficient of the current frame. The application equalization coefficient of the current frame is solved in combination with the equalization coefficient of the historical frame, and then the equalization coefficient of the current frame is stored in the equalization coefficient array of the historical frame. The application equalization coefficient of the current frame will not be stored in the equalization coefficient array of the historical frame, so as to ensure that the equalization coefficient of the historical frame is not affected by the output result of the brightness adjustment algorithm. In addition, when the length of the equalization coefficient array of the historical frame exceeds the set length, the oldest equalization coefficient is discarded and the latest equalization coefficient is stored.

[0164] Step S305: For any point other than the leftmost and rightmost sampling points in the horizontal direction of the fisheye camera image texture, the YUV channel equalization coefficient is set to 1, and for the points between the leftmost and rightmost sampling points, the YUV channel equalization coefficient increases or decreases arithmetically from the equalization coefficient corresponding to the leftmost point to the equalization coefficient corresponding to the rightmost point.

[0165] Step S306, according to the number of fisheye cameras, create a corresponding number of 3-channel M×N grid textures, and the 3 channels respectively store the equalization coefficients of the Y channel, U channel, and V channel. The application equalization coefficient obtained in step S305 is stored in the M×N grid texture, and all fisheye camera images are processed in the above manner to obtain the same number of M×N grid textures as the number of fisheye cameras, and the grid texture is used for color balance of the BEV stitched image.

[0166] Step S307, when performing BEV stitching, when sampling in the fisheye camera image through texture coordinates, the M×N grid texture is sampled using the same texture coordinates, so that for the YUV channel of each sampling point in the fisheye image, the corresponding application equalization coefficient can be found in the M×N grid texture, and finally the equalized RGB value is calculated.

[0167] Also, see Figure 4, and a schematic diagram of the overall process of an image processing method is also provided. In the overall processing flow, the original images collected by each fisheye camera are input into the brightness and color difference equalization module for processing; the brightness and color difference equalization module performs fusion area sampling; the weighted mean of each channel of the sampling point is calculated; a grid texture is created, and the texture content is the equalization coefficient after the smooth transition of each channel; and the grid texture is used to perform equalization processing on the original fisheye camera image. In addition, the BEV / 3D stitching module completes the stitching process of each image, and finally generates a panoramic image with brightness and color difference equalization processing.

[0168] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0169] This embodiment also provides an image processing device, which is used to implement the above embodiments and preferred implementations, and the descriptions that have been made will not be repeated. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0170] Figure 5 is a structural block diagram of an image processing device according to an embodiment of the present application, such as Figure 5 As shown, the device includes: an acquisition module 51, used to acquire multiple images to be processed; a fusion area sampling module 52, used to determine the fusion area of ​​each image to be processed and detect regional pixel points in the fusion area; an equalization coefficient calculation module 53, used to calculate the regional equalization coefficient of each image to be processed in the fusion area based on the regional pixel points, and calculate the transition equalization coefficient of each image to be processed according to the regional equalization coefficient and the position information of the pixel points in the image to be processed; a generation module 54, used to splice the images to be processed based on the transition equalization coefficient and generate a target panoramic image.

[0171] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination. For specific examples in this embodiment, reference can be made to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0172] This embodiment also provides a surround view system, including: an image acquisition device and a controller; the image acquisition device is used to acquire multiple images to be processed; the controller is connected to the image acquisition device, used to obtain each image to be processed, and execute the image processing method described in any of the above embodiments.

[0173] This embodiment also provides a vehicle, comprising the surround view system as described in the above embodiment.

[0174] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0175] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0176] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire multiple images to be processed; Determining a fusion region of each of the images to be processed, and detecting regional pixel points within the fusion region; Based on the regional pixels, a regional equalization coefficient in each fusion region on each side of the image to be processed is calculated, and according to the regional equalization coefficient and the position information of the pixels in the image to be processed, a transition equalization coefficient of each of the images to be processed is calculated; Based on the transition equalization coefficient, the images to be processed are spliced ​​to generate a target panoramic image.

2. The image processing method according to claim 1, characterized in that: The calculating, based on the regional pixel points, a regional equalization coefficient of each of the images to be processed in the fusion area comprises: For the fusion area, determining corresponding adjacent images from each of the images to be processed; the adjacent images include a first image and a second image; Based on the regional pixels, calculating a first average value of the first image within the fusion region, and based on the regional pixels, calculating a second average value of the second image within the fusion region; Calculating a total pixel value according to the first average value and the second average value; The regional equalization coefficient corresponding to the first image is calculated according to the first average value and the total pixel value, and the regional equalization coefficient corresponding to the second image is calculated according to the second average value and the total pixel value.

3. The image processing method according to claim 2, characterized in that: The calculating, based on the regional pixels, a first average value of the first image within the fusion region, and the calculating, based on the regional pixels, a second average value of the second image within the fusion region, comprises: Acquire chromaticity difference information between the first image and the second image; Based on the chromaticity difference information, assign a first weight value to a first pixel point in the first image corresponding to the pixel point in the region; perform weighted calculation on the pixel value of the first pixel point based on the first weight value to obtain the first average value; Based on the chromaticity difference information, a second weight value is assigned to a second pixel point in the second image corresponding to the pixel point in the area; and based on the second weight value, a weighted calculation is performed on the pixel value of the second pixel point to obtain the second average value.

4. The image processing method according to claim 1, characterized in that: The calculating, based on the regional pixel points, a regional equalization coefficient of each of the images to be processed in the fusion area comprises: Based on the regional pixel points, calculating the current frame equalization coefficient corresponding to the image to be processed; The regional equalization coefficient is calculated according to the current frame equalization coefficient and the acquired historical frame equalization coefficient.

5. The image processing method according to claim 1, characterized in that: The calculating, based on the regional pixel points, a regional equalization coefficient of each of the images to be processed in the fusion area comprises: Converting the image to be processed into a target color space where the brightness channel and the chrominance channel are separated; Calculate the regional equalization coefficient of the brightness channel and / or the chrominance channel of the image to be processed in the target color space.

6. The image processing method according to claim 5, characterized in that: The method further comprises: Detecting color difference information of the image to be processed; When it is detected that the color difference information is greater than a preset color difference threshold, a regional equalization coefficient of the image to be processed in the luminance channel and a regional equalization coefficient of the image to be processed in the chrominance channel are calculated respectively.

7. The image processing method according to claim 1, characterized in that: The fusion area in each of the images to be processed includes a first side area and a second side area; and calculating the transition equalization coefficient of each of the images to be processed according to the regional equalization coefficient and the position information of the pixel points in the images to be processed includes: Calculating a coefficient comparison result between a regional equalization coefficient of the first side area and a regional equalization coefficient of the second side area; Based on the coefficient comparison result, a corresponding transition equalization coefficient is allocated to each pixel point according to the position information of each pixel point in the image to be processed.

8. The image processing method according to claim 7, characterized in that: The step of allocating a corresponding transition equalization coefficient to each pixel point based on the coefficient comparison result and according to the position information of each pixel point in the image to be processed includes: Based on the position information of each pixel point, determine a first side pixel point and a second side pixel point in the image to be processed, and calculate a position comparison result between the first side pixel point and the second side pixel point; Calculating a unit allocation coefficient according to the coefficient comparison result and the position comparison result; Taking the regional equalization coefficient of the first side area as the starting point, the unit allocation coefficient is evenly and incrementally allocated to each pixel point according to the position difference between each pixel point and the first side pixel point, and the transition equalization coefficient is obtained.

9. The image processing method according to claim 1, characterized in that: The step of stitching the images to be processed based on the transition equalization coefficient and generating a target panoramic image includes: Get the preset grid texture image; Based on the position information of each pixel in the image to be processed, storing the transition equalization coefficient in the grid texture image; The grid texture image is used to calculate the equalization adjustment parameters for the image to be processed, and the image to be processed is spliced ​​based on the equalization adjustment parameters to generate the target panoramic image.

10. The image processing method according to any one of claims 1 to 9, characterized in that: The step of determining the fusion area of ​​each of the images to be processed includes: A virtual panoramic image is determined based on each of the images to be processed, and the fusion area in the virtual panoramic image is calculated.

11. The image processing method according to claim 10, characterized in that: The detecting of regional pixels within the fusion area comprises: Based on a preset shrinkage coefficient, shrinking the virtual panoramic image to obtain a pixel sampling area; Detect pixel points in the pixel sampling area that are located in the fusion area to obtain the regional pixel points.

12. The image processing method according to claim 11, characterized in that: The detecting pixel points in the pixel sampling area that are located in the fusion area to obtain the regional pixel points includes: Detecting an initial pixel point in the pixel sampling area that is located within the fusion area; Based on a preset sampling step size, the initial pixel points are sampled to obtain the regional pixel points.

13. An image processing device, characterized in that: include: An acquisition module, used for acquiring a plurality of images to be processed; A fusion region sampling module, used to determine the fusion region of each of the images to be processed, and detect regional pixel points within the fusion region; An equalization coefficient calculation module, used to calculate the regional equalization coefficient of each of the images to be processed in the fusion area based on the regional pixel points, and calculate the transition equalization coefficient of each of the images to be processed according to the regional equalization coefficient and the position information of the pixel points in the image to be processed; A generation module is used to splice the images to be processed based on the transition equalization coefficient and generate a target panoramic image.

14. A surround view system, characterized in that: include: Image acquisition equipment and controllers; The image acquisition device is used to acquire a plurality of images to be processed; The controller is connected to the image acquisition device, and is used to acquire each of the images to be processed and execute the image processing method according to any one of claims 1 to 12.

15. A vehicle, characterized in that: Comprising the surround view system as claimed in claim 14.

Images