Panoramic image processing method, device and equipment

Abstract

The embodiment of the invention provides a panoramic image processing method, device and equipment. The method comprises the following steps: acquiring single-side aerial views of different areas acquired by a vehicle-mounted camera; determining a composite aerial view based on the single-side aerial views, wherein the composite aerial view comprises an overlapping area of any two single-side aerial views; pixel correction is carried out on the composite aerial view based on a shielding processing algorithm; and carrying out pixel correction on the single-side aerial view based on the corrected composite aerial view, and outputting a panoramic image. The adjustment coefficient of the single-side aerial view can be obtained by correcting the composite aerial view, and then the single-side aerial view is corrected, so that the influence of a shielding object on the quality of the panoramic image can be effectively reduced, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of vehicle control, and particularly to a panoramic image processing method, apparatus, and device. Background Art

[0002] The panoramic image function is a common application of in-vehicle cameras, widely used for 360-degree monitoring of the vehicle's surrounding environment, helping the driver obtain a more comprehensive view, and assisting in decision-making. During the output of panoramic images, white balance processing is a key step for color-correcting images with color cast to obtain images with normal colors. However, due to the complex lighting conditions in daytime scenes, in-vehicle cameras are prone to occlusion (such as being blocked by obstacles, dirty lenses, etc.). Existing white balance adjustment methods cannot solve the occlusion problems encountered by vehicle cameras, and the quality of panoramic images will decline, affecting the user experience. Summary of the Invention

[0003] In view of this, this application provides a panoramic image processing method, apparatus, and device to facilitate solving the problem that the quality of panoramic images deteriorates due to obstacles blocking in-vehicle cameras in the prior art.

[0004] In a first aspect, an embodiment of this application provides a panoramic image processing method, including:

[0005] Obtain single-sided bird's-eye views of different regions collected by an in-vehicle camera;

[0006] Determine a composite bird's-eye view based on the single-sided bird's-eye views, where the composite bird's-eye view includes the overlapping regions of any two single-sided bird's-eye views;

[0007] Perform pixel correction on the composite bird's-eye view based on an occlusion processing algorithm;

[0008] Perform pixel correction on the single-sided bird's-eye views based on the corrected composite bird's-eye view, and output a panoramic image.

[0009] In an optional embodiment, the performing pixel correction on the composite bird's-eye view based on an occlusion processing algorithm includes:

[0010] In response to the in-vehicle camera being blocked by a colored obstacle, perform color cast correction processing on the pixel values of the composite bird's-eye view based on a first occlusion processing algorithm;

[0011] In response to the in-vehicle camera being blocked by a black obstacle, perform occlusion correction processing on the pixel values of the composite bird's-eye view based on a second occlusion processing algorithm.

[0012] In an optional embodiment, the performing color cast correction processing on the pixel values of the composite bird's-eye view based on a first occlusion processing algorithm includes:

[0013] Judge whether there is a color deviation phenomenon in the composite bird's-eye view based on the pixel mean values of each color channel of the composite bird's-eye view;

[0014] When it is determined that there is a color deviation phenomenon in the composite bird's-eye view, perform gray-scale conversion on the pixel values of each color channel of the composite bird's-eye view. When it is determined that there is no color deviation phenomenon in the composite bird's-eye view, do not process the pixel values of each color channel of the composite bird's-eye view.

[0015] In an optional embodiment, the occlusion correction process for the pixel values of the composite bird's-eye view based on the second occlusion processing algorithm includes:

[0016] Divide each composite bird's-eye view into a dark area or a non-dark area based on the gray-scale values of each composite bird's-eye view;

[0017] For any composite bird's-eye view divided into a dark area, correct the gray-scale value of the current composite bird's-eye view based on the gray-scale values of adjacent composite bird's-eye views.

[0018] In an optional embodiment, the correction process for the gray-scale value of the current composite bird's-eye view based on the gray-scale values of adjacent composite bird's-eye views includes:

[0019] If both of the two composite bird's-eye views adjacent to the current composite bird's-eye view are non-dark areas, determine the mean value of the gray-scale values of the current composite bird's-eye view and the two adjacent composite bird's-eye views as the gray-scale value of the current composite bird's-eye view;

[0020] If any of the composite bird's-eye views adjacent to the current composite bird's-eye view is a non-dark area, determine the mean value of the gray-scale values of the current composite bird's-eye view and the adjacent composite bird's-eye view classified as a non-dark area as the gray-scale value of the current composite bird's-eye view.

[0021] In an optional embodiment, the pixel correction of the single-sided bird's-eye view based on the corrected composite bird's-eye view and the output of the panoramic image include:

[0022] Construct error equations for each color channel based on the corrected composite bird's-eye view;

[0023] Solve the error equations for each color channel to obtain the adjustment coefficients for each color channel of each single-sided bird's-eye view;

[0024] Correct the pixel values of the single-sided bird's-eye view based on the adjustment coefficients and output the panoramic image.

[0025] In an optional embodiment, the correction of the pixel values of the single-sided bird's-eye view based on the adjustment coefficients and the output of the panoramic image include:

[0026] Multiply the pixel values of the single-sided bird's-eye view by the corresponding adjustment coefficients to obtain the corrected single-sided bird's-eye view;

[0027] Fuse the overlapping regions of the corrected single-sided bird's-eye view based on the fusion algorithm to output the panoramic image.

[0028] In a second aspect, an embodiment of the present application provides a panoramic image processing device, including:

[0029] An acquisition module, configured to acquire single-sided bird's-eye views of different regions collected by a vehicle-mounted camera;

[0030] A determination module, configured to determine a composite bird's-eye view based on the single-sided bird's-eye view, where the composite bird's-eye view includes overlapping regions of any two single-sided bird's-eye views;

[0031] A first correction module, configured to perform pixel correction on the composite bird's-eye view based on an occlusion processing algorithm;

[0032] A second correction module, configured to perform pixel correction on the single-sided bird's-eye view based on the corrected composite bird's-eye view and output a panoramic image.

[0033] In a third aspect, an embodiment of the present application provides an electronic device, including a memory for storing computer program instructions and a processor for executing the program instructions. When the computer program instructions are executed by the processor, the electronic device is triggered to execute the method according to any one of the first aspects above.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium includes a stored program. When the program runs, it controls the device where the computer-readable storage medium is located to execute the method according to any one of the first aspects.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, where the computer program product includes executable instructions. When the executable instructions are executed on a computer, the computer is caused to execute the method according to any one of the first aspects.

[0036] Adopting the solution provided by the embodiment of the present application, single-sided bird's-eye views of different regions collected by a vehicle-mounted camera are acquired; a composite bird's-eye view is determined based on the single-sided bird's-eye view, and the composite bird's-eye view includes overlapping regions of any two single-sided bird's-eye views; pixel correction is performed on the composite bird's-eye view based on an occlusion processing algorithm; pixel correction is performed on the single-sided bird's-eye view based on the corrected composite bird's-eye view, and a panoramic image is output. By correcting the composite bird's-eye view, an adjustment coefficient of the single-sided bird's-eye view can be obtained, and then the single-sided bird's-eye view can be corrected, which can effectively reduce the influence of occlusions on the quality of the panoramic image and improve the user experience. Description of the Drawings

[0037] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0038] Figure 1 It is a schematic flowchart of a panoramic image processing method provided by an embodiment of the present application;

[0039] Figure 2 It is a schematic example diagram of a panoramic image processing method provided by an embodiment of the present application;

[0040] Figure 3 It is a schematic example diagram of another panoramic image processing method provided by an embodiment of the present application;

[0041] Figure 4 It is a schematic flowchart of another panoramic image processing method provided by an embodiment of the present application;

[0042] Figure 5 It is a schematic example diagram of another panoramic image processing method provided by an embodiment of the present application;

[0043] Figure 6 It is a schematic example diagram of another panoramic image processing method provided by an embodiment of the present application;

[0044] Figure 7 It is a schematic flowchart of another panoramic image processing method provided by an embodiment of the present application;

[0045] Figure 8 It is a schematic flowchart of another panoramic image processing method provided by an embodiment of the present application;

[0046] Figure 9 It is a schematic structural diagram of a panoramic image processing device provided by an embodiment of the present application;

[0047] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0048] To better understand the technical solutions of the present application, the following will describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0049] It should be clear that the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0050] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "the" and "said" used in the embodiments of the present application and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise.

[0051] It should be understood that the term " / and" used herein is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A / and B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " herein generally indicates that the associated objects before and after are in an "or" relationship.

[0052] With the rapid development of autonomous driving technology, the panoramic imaging function has gradually become an essential function of vehicles. A plurality of on-vehicle cameras are arranged around the vehicle to monitor the surrounding environment of the vehicle 360 degrees, and a panoramic image is output on the in-vehicle screen to help the driver obtain a more comprehensive view and assist in decision-making. During the output of the panoramic image, white balance adjustment is a key step. Existing technologies usually rely on methods based on physical models or statistical learning to adjust the white balance by calculating the color information in the image, so as to eliminate the color deviation under different lighting conditions. Most existing white balance algorithms have been able to handle ambient light changes to a certain extent, but in practical applications, there are at least the following problems:

[0053] (1) Obstacle occlusion: The on-vehicle cameras often encounter occlusions (such as obstacles, dirty lenses, etc.) in daytime scenes. These occlusions will cause large-area color blocks to appear in the image and be accompanied by uneven light distribution, affecting the accuracy of white balance adjustment, and further affecting the quality of the panoramic image and subsequent automated analysis and decision-making.

[0054] (2) Algorithm complexity and efficiency issues: Existing white balance algorithms usually require high-performance computing platforms and complex mathematical libraries. For example, image processing methods based on deep learning train a convolutional neural network (CNN) model, learn image features using a large amount of data, and then predict and adjust the white balance. The disadvantage of such methods is that they require a large amount of computing resources and training data, rely on advanced hardware support, and have a high input cost.

[0055] In view of the above problems, the embodiments of the present application provide a panoramic image processing method, which determines an adjustment coefficient based on a composite bird's-eye view of an overlapping area, corrects a unilateral bird's-eye view, and then outputs a high-quality panoramic image.

[0056] Figure 1 It is a schematic flowchart of a panoramic image processing method provided by an embodiment of the present application. This method can be executed by a processing device installed in a vehicle, such as Figure 1 shown, this method may include:

[0057] Step 101: Obtain single-sided bird's-eye views of different areas collected by in-vehicle cameras.

[0058] Step 102: Determine a composite bird's-eye view based on the single-sided bird's-eye views. The composite bird's-eye view includes the overlapping areas of any two single-sided bird's-eye views.

[0059] Step 103: Perform pixel correction on the composite bird's-eye view based on an occlusion processing algorithm.

[0060] Step 104: Perform pixel correction on the single-sided bird's-eye views based on the corrected composite bird's-eye view and output a panoramic image.

[0061] Multiple in-vehicle cameras are installed in the vehicle. The images collected by each in-vehicle camera can be regarded as a single-sided bird's-eye view. Optionally, the in-vehicle cameras may include a front camera, a rear camera, a left camera, and a right camera. Refer to Figure 2 , the image collected by the front camera is a front-side bird's-eye view, the image collected by the rear camera is a rear-side bird's-eye view, the image collected by the left camera is a left-side bird's-eye view, and the image collected by the right camera is a right-side bird's-eye view. Among them, the front-side bird's-eye view has overlapping areas with the left-side bird's-eye view and the right-side bird's-eye view respectively, and the rear-side bird's-eye view also has overlapping areas with the left-side bird's-eye view and the right-side bird's-eye view respectively. Refer to Figure 3 , each single-sided bird's-eye view contains two overlapping areas, a total of 8 overlapping areas, and each overlapping area can be regarded as a composite bird's-eye view.

[0062] Generally, the occlusion of in-vehicle cameras by obstacles may include the following situations: (1) Color occlusion: such as a colored obstacle being close to the in-vehicle camera, a translucent colored obstacle sticking to the camera, or a large area of colored content in the world coordinates, etc. These situations cause the camera to capture a large area of colored blocks. (2) Black occlusion: such as an opaque obstacle occupying most of the field of view of the in-vehicle camera, an opaque obstacle covering the in-vehicle camera, or a large area of black content in the world coordinates, etc. These situations cause the camera to capture a large area of black shadows.

[0063] For the phenomenon of colored obstacle occlusion of in-vehicle cameras, the processing device can perform color deviation correction processing on the pixel values of the composite bird's-eye view based on the first occlusion processing algorithm. For the phenomenon of black obstacle occlusion of in-vehicle cameras, the processing device can perform occlusion correction processing on the pixel values of the composite bird's-eye view based on the second occlusion processing algorithm.

[0064] (1) Color deviation processing: After determining the composite bird's-eye view corresponding to each single-sided bird's-eye view, the processing device can judge whether there is a color deviation phenomenon in the composite bird's-eye view based on the pixel mean values of each color channel of the composite bird's-eye view. When it is determined that there is a color deviation phenomenon in the composite bird's-eye view, the pixel values of each color channel of the composite bird's-eye view are subjected to grayscale conversion. When it is determined that there is no color deviation phenomenon in the composite bird's-eye view, the pixel values of each color channel of the composite bird's-eye view are not processed.

[0065] Specifically, define the width of each composite bird's-eye view as W, the height as H, the pixel point coordinates as (x, y), and meanR, meanG, and meanB as the pixel mean values of the R channel, G channel, and B channel respectively. The processing device can calculate the channel mean values of 8 composite bird's-eye views respectively, and the calculation formula is as follows:

[0066]

[0067] Define T1, T2, T3, and T4 as the thresholds of the gray system within the range of 2500K - 7500K of color temperature. If the ratio of the channel mean values of the current overlapping image is not within the range of the gray system thresholds, it is judged as color deviation, that is, there are large areas of colored blocks in the overlapping image. At this time, the channel mean values are subjected to grayscale conversion, and the converted values are constrained within the range of 1 - 255. The specific steps can refer to Figure 4 , mainly including:

[0068] Step 401, input the composite bird's-eye view channel mean values meanR, meanG, and meanB.

[0069] Step 402, judge whether each channel mean value meets the conditions T1 < (meanR / meanG) < T2 and T3 < (meanB / meanG) < T4? If so, enter step 404, otherwise enter step 403.

[0070] Step 403, perform grayscale conversion on the pixel values of the composite bird's-eye view.

[0071] Step 404, output the composite bird's-eye view.

[0072] In the embodiment of the present application, for the area with color occlusion, the processing device converts the color information in the composite bird's-eye view into brightness information to weaken the influence of color on adjacent areas.

[0073] (2) Occlusion processing: After determining the composite bird's-eye view corresponding to each single-sided bird's-eye view, the processing device can divide each composite bird's-eye view into a dark area or a non-dark area based on the grayscale value of each composite bird's-eye view. For any composite bird's-eye view divided into a dark area, the processing device corrects the grayscale value of the current composite bird's-eye view based on the grayscale value of the adjacent composite bird's-eye view.

[0074] Since there is an overlapping area between the images collected by two vehicle-mounted cameras, if the image of one vehicle-mounted camera is abnormal due to obstacle occlusion, the abnormal image can be pixel-corrected with reference to the overlapping area. Refer to Figure 5 , the overlapping part between the left bird's-eye view and the front bird's-eye view in the left side is called the left-front bird's-eye view, and the overlapping part between the front bird's-eye view and the left bird's-eye view is called the front-left bird's-eye view, and the same applies to others. The left-front bird's-eye view and the front-left bird's-eye view are images collected by different vehicle-mounted cameras for the same area. If the front camera is occluded and the front-left bird's-eye view is abnormal, the processing device can correct the front-left bird's-eye view based on the left-front bird's-eye view.

[0075] Specifically, the processing device calculates the gray-scale mean value of each composite bird's-eye view, and the calculation formula is as follows:

[0076] Grey=0.114*meanR+0.578*meanG+0.229*meanB

[0077] The processing device marks the areas with gray-scale mean value less than the gray-scale threshold as dark areas, and those greater than or equal to the gray-scale threshold as non-dark areas. Optionally, the gray-scale threshold can be set based on experience, such as set to values like 50, 60, etc. The average brightness of the dark areas is low, and it is highly probable to belong to black occlusion. The average brightness of the non-dark areas is high and can be regarded as normal images. Connect the above 8 composite bird's-eye views in series in the order in the panoramic view to obtain a series-connected relationship diagram with the head and tail connected, as Figure 6 shown.

[0078] The processing device can process the composite bird's-eye view based on the following rules:

[0079] (1) If the number of non-dark areas is 0, that is, the average brightness of all composite bird's-eye views is low and the overall brightness of the panoramic view is balanced, no processing is required.

[0080] (2) If the adjacent composite bird's-eye views before and after a dark area are both non-dark areas, then add the gray-scale values of the dark area and the adjacent composite bird's-eye views before and after and take the average value, and use this average value to replace the original gray-scale value of the dark area.

[0081] (3) If the adjacent composite bird's-eye view before or after a dark area is non-dark, then add the gray-scale values of the dark area and the adjacent bird's-eye view before or after and take the average value, and use this average value to replace the original gray-scale value of the dark area.

[0082] (4) If the adjacent composite bird's-eye views before and after a dark area are also both dark areas, no processing is required for the current dark area.

[0083] The specific steps can be referred to the Figure 7 shown flowchart, which mainly includes:

[0084] Step 701, construct a series relationship diagram based on the composite bird's-eye view.

[0085] Step 702, determine the dark area and non-dark area based on the gray mean value of the composite bird's-eye view.

[0086] Step 703, judge whether the number of non-dark areas is greater than 0. If so, go to Step 704; otherwise, go to Step 710.

[0087] Step 704, under the condition that the i-th composite bird's-eye view is a dark area, judge whether both the (i + 1)-th and (i - 1)-th composite bird's-eye views are non-dark areas. If so, go to Step 707; otherwise, go to Step 705.

[0088] Step 705, under the condition that the i-th composite bird's-eye view is a dark area, judge whether the (i + 1)-th composite bird's-eye view is a non-dark area. If so, go to Step 708; otherwise, go to Step 706.

[0089] Step 706, under the condition that the i-th composite bird's-eye view is a dark area, judge whether the (i - 1)-th composite bird's-eye view is a non-dark area. If so, go to Step 709; otherwise, go to Step 707.

[0090] Step 707, determine the gray value of the i-th composite bird's-eye view as the mean value of the gray values of the (i - 1)-th, i-th, and (i + 1)-th composite bird's-eye views.

[0091] Step 708, determine the gray value of the i-th composite bird's-eye view as the mean value of the gray values of the i-th and (i + 1)-th composite bird's-eye views.

[0092] Step 709, determine the gray value of the i-th composite bird's-eye view as the mean value of the gray values of the i-th and (i - 1)-th composite bird's-eye views.

[0093] Step 710, output the gray value of the composite bird's-eye view.

[0094] Through the above process, the processing device can reduce the brightness difference between the dark area and the non-dark area and weaken the spread of darkness in adjacent areas.

[0095] After the color deviation processing and occlusion processing are completed, the processing device can construct an error equation for each color channel based on the corrected composite bird's-eye view. Solving the error equation for each color channel can obtain the adjustment coefficient for each color channel of each unilateral bird's-eye view. Based on this adjustment coefficient, the pixel values of the unilateral bird's-eye view are corrected, and a panoramic image with higher quality can be output.

[0096] Specifically, the panoramic image is composed of n single-sided bird's-eye view images. The composite bird's-eye view image is represented by the digital combination "ij", where both i and j belong to n, and i and j correspond to different single-sided bird's-eye view images. In the embodiments of the present application, the front bird's-eye view image is defined as 1, the left bird's-eye view image is defined as 2, the right bird's-eye view image is defined as 3, and the rear bird's-eye view image is defined as 4. Then the definition of the composite bird's-eye view image is as follows:

[0097] (1) The front-left bird's-eye view image is 12, and the front-right bird's-eye view image is 13;

[0098] (2) The left-front bird's-eye view image is 21, and the left-rear bird's-eye view image is 24;

[0099] (3) The right-front bird's-eye view image is 31, and the right-rear bird's-eye view image is 34;

[0100] (4) The rear-left bird's-eye view image is 42, and the rear-right bird's-eye view image is 43.

[0101] The total number of pixels of the composite bird's-eye view image "ij" is defined as Num. One overlapping area can construct 1 error equation according to the composite bird's-eye view image "ij" and the composite bird's-eye view image "ji", and four overlapping areas can construct 4 error equations. After organizing and summarizing all the error equations, the following expression is finally obtained:

[0102]

[0103] Among them, e represents the minimum error, represents the adjustment coefficient of the channel of the single-sided bird's-eye view image i, represents the adjustment coefficient of the channel of the single-sided bird's-eye view image j, represents the pixel mean value of the channel of the composite image "ij", represents the pixel mean value of the channel of the composite bird's-eye view image "ji", channel ∈ (R, G, B). σ N is the color error, generally set to 10.0, σ g is the empirical value of the standard deviation of the gain parameter, generally set to 0.1. According to the above error equation formula, set the partial derivative of the minimum error to 0, and after organizing, the following error equation group expression is obtained, in the form of:

[0104]

[0105] Using the idea of elimination and gradually organizing, the expression about the adjustment coefficient can be finally obtained, in the form of:

[0106]

[0107] Among them, φ index 、μ index 、ω index 、bindex , λ index are all constant terms, where index ∈ [1, 16]. Substitute the data after image color deviation processing or occlusion processing into the expression regarding the adjustment coefficient . Only four arithmetic operations are needed to solve each adjustment coefficient.

[0108] The processing device multiplies the pixel values of the single-sided bird's-eye view by the corresponding adjustment coefficients to obtain the corrected single-sided bird's-eye view. Then, based on the fusion algorithm, the corrected single-sided bird's-eye view is fused to output the panoramic image.

[0109] Define as the channel of the i-th single-sided bird's-eye view, and as the channel of the adjusted i-th single-sided bird's-eye view, where i ∈ [1, 4]. The adjustment formula for the single-sided bird's-eye view is:

[0110]

[0111] Use any fusion algorithm (such as alpha fusion) to fuse the two composite bird's-eye views in each overlapping area to achieve the effect of natural overlap of the two composite bird's-eye views. The common formula for alpha fusion is as follows:

[0112] C overlap = αC ij + (1 - α)C ji

[0113] where C overlap is defined as the fused image, C ij and C ji are respectively defined as the overlapping image "ij" and the overlapping image "ji", and α is defined as the fusion factor, where α ∈ [0, 1]. Place the adjusted single-sided bird's-eye view and the fused composite bird's-eye view on the corresponding coordinates to complete the panoramic view stitching, and finally output a high-quality panoramic image with balanced color and brightness.

[0114] Figure 8 is a schematic flowchart of another panoramic image processing method provided by an embodiment of the present application. As Figure 8 shown, the method may include:

[0115] Step 801, input the front, rear, left, and right single-sided bird's-eye views.

[0116] Step 802, determine the composite bird's-eye view and calculate the channel mean value.

[0117] Step 803, based on the channel mean value, determine whether it belongs to the color deviation phenomenon. If so, go to step 805; otherwise, go to step 804.

[0118] Step 804 , judging whether it is an occlusion phenomenon based on the channel mean value, if so, proceeding to step 806 , otherwise proceeding to step 807 .

[0119] Step 805: Perform color shift processing on the composite bird's-eye view image.

[0120] Step 806, performing occlusion processing on the composite bird's-eye view.

[0121] Step 807: construct an error equation to solve the adjustment coefficient.

[0122] Step 808: correct the single-sided bird's-eye view based on the adjustment coefficient and output a panoramic image.

[0123] The color cast phenomenon is the color occlusion phenomenon mentioned above, and the occlusion phenomenon is the black occlusion phenomenon mentioned above.

[0124] In the embodiment of the present application, the processing device performs two aspects of processing on the composite bird's-eye view: the color deviation processing ensures that the image color is not distorted during the white balance adjustment process, and the occlusion processing reduces the influence of black shadows on the global brightness during the white balance adjustment process, greatly improving the accuracy of the white balance adjustment of the surround view. The above processes can all be executed on chips and processors with lower performance, without relying on special math libraries and complex hardware acceleration devices. The solution of the minimum error equation group can be completed through the most basic arithmetic operations (addition, subtraction, multiplication, and division), avoiding high-cost operations such as matrix operations and vector operations. The image color deviation judgment and occlusion judgment are performed through the channel mean of the composite bird's-eye view, without the need to traverse and screen pixel points one by one, which can meet the real-time requirements and ensure the accuracy and stability of image processing in an environment with limited memory and computing power.

[0125] Figure 9 Schematic diagram of the structure of a panoramic image processing device provided in an embodiment of the present application. Figure 9 As shown, the device may include:

[0126] The acquisition module 910 is used to acquire the single-side bird's-eye view of different areas collected by the vehicle-mounted camera.

[0127] The determination module 920 is used to determine a composite bird's-eye view based on the single-side bird's-eye view, where the composite bird's-eye view includes an overlapped area of ​​any two single-side bird's-eye views.

[0128] The first correction module 930 is used to perform pixel correction on the composite bird's-eye view image based on an occlusion processing algorithm.

[0129] The second correction module 940 is used to perform pixel correction on the single-side bird's-eye view based on the corrected composite bird's-eye view and output a panoramic image.

[0130] Corresponding to the above embodiments, the present application also provides an electronic device. Figure 10A schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 1000 may include: a processor 1001, a memory 1002, and a communication unit 1003. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiments of the present application. It can be a bus structure, a star structure, and may also include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0131] Among them, the communication unit 1003 is used to establish a communication channel so that the electronic device can communicate with other devices. Receive user data sent by other devices or send user data to other devices.

[0132] The processor 1001 is the control center of the electronic device. It uses various interfaces and lines to connect all parts of the entire electronic device. By running or executing software programs, instructions, and / or modules stored in the memory 1002, and by calling data stored in the memory, it executes various functions of the electronic device and / or processes data. The processor may be composed of an integrated circuit (IC). For example, it may be composed of a single packaged IC, or may be composed of multiple packaged ICs with the same or different functions connected together. For example, the processor 1001 may only include a central processing unit (CPU). In the embodiment of the present application, the CPU may be a single arithmetic core or may include multiple arithmetic cores.

[0133] The memory 1002 is used to store the execution instructions of the processor 1001. The memory 1002 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0134] When the execution instructions in the memory 1002 are executed by the processor 1001, the electronic device 1000 can execute some or all of the steps in the above embodiments.

[0135] In specific implementation, the present application further provides a computer storage medium. The computer storage medium can store a program, and when the program is executed, it may include some or all of the steps in the embodiments of the panoramic image processing method provided by the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), or the like.

[0136] In specific implementation, the present application further provides a computer program product. The computer program product includes executable instructions, and when the executable instructions are executed on a computer, the computer is caused to execute some or all of the steps in the embodiments of the panoramic image processing method provided by the present application.

[0137] The embodiments of the present application further provide a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the panoramic image processing method provided by the embodiments of the present application.

[0138] The above non-transitory computer-readable storage medium may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0139] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take many forms, including - but not limited to - electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0140] The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including - but not limited to - wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0141] Those skilled in the art can clearly understand that the technologies in the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present application.

[0142] For the same or similar parts among the various embodiments in this specification, reference can be made to each other. In particular, for the apparatus embodiments and terminal embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A panoramic image processing method, characterized in that: include: Get a bird's-eye view of different areas captured by the on-board camera; Determine a composite bird's-eye view based on the single-sided bird's-eye view, where the composite bird's-eye view includes an overlapping area of ​​any two single-sided bird's-eye views; Performing pixel correction on the composite bird's-eye view image based on an occlusion processing algorithm; The single-side bird's-eye view is pixel-corrected based on the corrected composite bird's-eye view, and a panoramic image is output.

2. The method according to claim 1, characterized in that The pixel correction of the composite bird's-eye view image based on the occlusion processing algorithm includes: In response to the vehicle-mounted camera being blocked by a colored obstacle, performing color deviation correction processing on pixel values ​​of the composite bird's-eye view image based on a first blocking processing algorithm; In response to the vehicle-mounted camera being blocked by a black obstacle, occlusion correction processing is performed on the pixel values ​​of the composite bird's-eye view based on a second occlusion processing algorithm.

3. The method according to claim 2, characterized in that The color deviation correction processing of the pixel values ​​of the composite bird's-eye view image based on the first occlusion processing algorithm includes: Determining whether the composite bird's-eye view image has a color cast phenomenon based on the pixel mean value of each color channel of the composite bird's-eye view image; When it is determined that the composite bird's-eye view has the color cast phenomenon, the pixel values ​​of each color channel of the composite bird's-eye view are converted into grayscale; when it is determined that the composite bird's-eye view does not have the color cast phenomenon, the pixel values ​​of each color channel of the composite bird's-eye view are not processed.

4. The method according to claim 2, characterized in that: The performing occlusion correction processing on the pixel values ​​of the composite bird's-eye view image based on the second occlusion processing algorithm comprises: Dividing each composite bird's-eye view image into a dark area or a non-dark area based on the grayscale value of each composite bird's-eye view image; For any composite bird's-eye view image that is divided into a dark area, the grayscale value of the current composite bird's-eye view image is corrected based on the grayscale values ​​of adjacent composite bird's-eye view images.

5. The method according to claim 4, characterized in that The step of correcting the grayscale value of the current composite bird's-eye view based on the grayscale value of the adjacent composite bird's-eye view includes: If the two composite bird's-eye view images adjacent to the current composite bird's-eye view image are both non-dark areas, then the average of the grayscale values ​​of the current composite bird's-eye view image and the two adjacent composite bird's-eye view images is determined as the grayscale value of the current composite bird's-eye view image; If any composite bird's-eye view adjacent to the current composite bird's-eye view is a non-dark area, the average of the grayscale values ​​of the current composite bird's-eye view and the adjacent composite bird's-eye view classified as a non-dark area is determined as the grayscale value of the current composite bird's-eye view.

6. The method according to claim 1, characterized in that The pixel correction of the single-side bird's-eye view based on the corrected composite bird's-eye view and outputting a panoramic image includes: constructing error equations for each color channel based on the corrected composite bird's-eye view; Solving the error equations of the color channels to obtain adjustment coefficients of the color channels of each single-sided bird's-eye view; The pixel values ​​of the single-side bird's-eye view are corrected based on the adjustment coefficient, and a panoramic image is output.

7. The method according to claim 6, characterized in that The step of correcting the pixel value of the single-side bird's-eye view image based on the adjustment coefficient and outputting a panoramic image includes: Multiplying the pixel value of the one-sided bird's-eye view by the corresponding adjustment coefficient to obtain a corrected one-sided bird's-eye view; Based on a fusion algorithm, the overlapping areas of the corrected single-sided bird's-eye view are fused to output the panoramic image.

8. A panoramic image processing device, characterized in that: include: An acquisition module, used to obtain a single-side bird's-eye view of different areas collected by the vehicle-mounted camera; A determination module, configured to determine a composite bird's-eye view based on the single-side bird's-eye view, wherein the composite bird's-eye view includes an overlapped area of ​​any two single-side bird's-eye views; A first correction module, used for performing pixel correction on the composite bird's-eye view image based on an occlusion processing algorithm; The second correction module is used to perform pixel correction on the single-side bird's-eye view based on the corrected composite bird's-eye view and output a panoramic image.

9. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Images