Compensation method for color difference between adjacent lens images in panoramic images

Abstract

The present invention discloses a method for compensating color difference between adjacent lens images in a panoramic image, which is used to compensate color difference between adjacent lens images in a panoramic image, each lens image includes multiple feature areas and each feature area has a feature point. The method includes: calculating the color difference value of each pair of feature areas between any two adjacent lens images in multiple lens images generated by a camera with multiple lenses according to the average color value of the feature points; determining the color adjustment amount of the feature points in each lens image according to the color difference value of each pair of feature areas; and calculating the color adjustment amount of a unit according to the positions of two feature points adjacent to the unit and the color adjustment amounts of the two adjacent feature points. By using the present invention, the image quality under the color difference phenomenon can be improved.

Description

Technical Field

[0001] The present invention relates to panoramic images, and more particularly to a method for compensating color differences between adjacent lens images in a panoramic image. Background Art

[0002] A multi-lens camera captures a field of view that covers 360 degrees horizontal field of view (FOV) and up to 180 degrees vertical field of view. Figure 1A As shown, a conventional camera includes four lenses and is respectively mounted on four working surfaces (right, left, front, and back) of a cubic structure 11. Any two adjacent surfaces of the four working surfaces are substantially orthogonal. For example, the four working surfaces are respectively oriented toward the longitudes of 0, 90, 180, and 270 of a virtual sphere 12 to simultaneously capture a field of view with a 360-degree horizontal FOV and a 90-degree vertical FOV to generate four lens images. A necessary condition for setting up the four-lens camera is that the fields of view of any two adjacent lenses should have sufficient overlap to facilitate image mosaicking. Please refer to Figure 1B , the pixels in the area 13 are formed by the overlap of two lens / texture images, while the pixels in the area 15 are from a single lens / texture image. Therefore, the overlapping area 13 can be stitched to form a panoramic image.

[0003] In a very likely situation, if one of the lenses of the camera with four lenses is directly facing a light source (such as the sun) and the other lens is facing a dark place, even with the same exposure time, the color performance of the images (including the overlapping area 13) produced by the two lenses will be very different, which is hereinafter referred to as "color difference phenomenon". Therefore, the industry urgently needs a compensation method to improve the image quality under the above color difference phenomenon. Summary of the invention

[0004] In view of the above problems, one of the objectives of the present invention is to provide a compensation method to improve the image quality under the color difference phenomenon.

[0005] According to an embodiment of the present invention, a compensation method is provided for compensating the color difference between adjacent lens images in a panoramic image, each lens image including a plurality of feature areas and each feature area having a feature point. The method comprises: calculating the color difference value of each pair of feature areas between any two adjacent lens images in a plurality of lens images generated by a camera with multiple lenses according to the average color value of the feature points; determining the color adjustment amount of the feature points in each lens image according to the color difference value of each pair of feature areas; and calculating the color adjustment amount of a unit according to the positions of two feature points adjacent to the unit and the color adjustment amounts of the two adjacent feature points.

[0006] By using the present invention, the image quality under the color difference phenomenon can be improved.

[0007] The above and other objects and advantages of the present invention are described in detail below with reference to the following drawings, detailed description of the embodiments and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0009] Figure 1A The diagram shows the relationship between a cubic structure 11 and a sphere 12 .

[0010] Figure 1B It displays an equirectangular panoramic image, which is derived from the equirectangular projection of four lens images (back, left, right, and front).

[0011] Figure 2 FIG. 4 is a block diagram showing a panoramic image processing system according to an embodiment of the present invention.

[0012] Figure 3A FIG. 4 is a flow chart of a method for displaying the average color values ​​of all pixels in each feature area according to the first embodiment of the present invention.

[0013] Figure 3B It is an example showing that the edges of a texture / lens image are divided into three blocks relative to the image center O.

[0014] Figure 4A Displays a triangular mesh modeling the surface of a sphere.

[0015] Figure 4BIt displays a polygonal mesh used to compose / model an equirectangular panoramic image.

[0016] Figure 5A FIG. 4 is a flow chart of a method for displaying the average color values ​​of all pixels in each feature area according to a second embodiment of the present invention.

[0017] Figure 5B An example of displaying a plurality of selected values ​​as the longitude and latitude coordinates of eight centroids of eight feature regions of each texture / lens image (back, left, right, front).

[0018] Fig. 6A It shows the mismatched stitching results due to the offset lens center.

[0019] Figure 6B This is an example of displaying an equidistant rectangular panoramic image with twelve overlapping areas according to the prior art.

[0020] Figure 6C This is an example of displaying an equidistant rectangular panoramic image with twenty control areas according to the prior art.

[0021] Fig.6D FIG. 1 is an example showing the relationship between the target vertex P2 and the eight control regions in a frontal lens image.

[0022] Figure 7 FIG. 4 is a block diagram showing an image processing device according to an embodiment of the present invention.

[0023] Fig. 8A An example of three job pairs showing the texture / lens images for job 0 and job 1.

[0024] Figure 8B 4 is a flow chart of a method for compensating color differences between adjacent lens images in a panoramic image according to a first embodiment of the present invention.

[0025] Figure 8C An example is shown, in which the eight initial color adjustment amounts of the eight feature areas on the left are low-pass filtered using the low-pass filter on the right to obtain the eight final color adjustment amounts below.

[0026] Fig.8D An example of the positional relationship between an image center O, a target vertex / pixel S, and two feature points (G and H) adjacent to the target vertex / pixel S in a frontal lens image is shown.

[0027] Fig. 8E FIG. 4 is a flow chart of a method for compensating color differences between adjacent lens images in a panoramic image according to a second embodiment of the present invention.

[0028] Fig.9A The present invention is a flowchart of a method for obtaining color adjustment amounts and radii of a plurality of feature points in all lens images in a panoramic image according to an embodiment of the present invention.

[0029] FIG. 9B to FIG. 9C FIG. 4 is a flow chart of a method for compensating color differences between adjacent lens images in a panoramic image according to a third embodiment of the present invention.

[0030] 11: Cube Architecture

[0031] 12:Sphere

[0032] 13: Overlapping area

[0033] 15: Non-overlapping area

[0034] 32: Best focus

[0035] 33: Ideal imaging position

[0036] 35: Distant Objects

[0037] 36, 37: Actual focus

[0038] 38: Actual imaging position

[0039] 62: Location of the control zone

[0040] 72: Texture mapping circuit

[0041] 73: Mixing unit

[0042] 74: Destination Buffer

[0043] 75: Rasterization Engine

[0044] 77: Measuring unit

[0045] 200: Panoramic image processing system

[0046] 210: Image capture module

[0047] 220: Image processing device

[0048] 250: Correspondence Generator

[0049] 710: Computing unit

[0050] 721~72N: Texture Mapping Engine

[0051] 750: Imaging Engine DETAILED DESCRIPTION

[0052] The singular terms "a", "an", "the" and the like mentioned throughout the specification and the subsequent claims include both the singular and the plural, unless otherwise specified in this specification. The relevant terms mentioned throughout the specification and the subsequent claims are defined as follows, unless otherwise specified in this specification. The relevant terms mentioned throughout the specification and the subsequent claims are defined as follows, unless otherwise specified in this specification. Throughout the specification, circuit elements with the same function use the same reference symbols.

[0053] Figure 2 2 is a block diagram showing a panoramic image processing system according to an embodiment of the present invention. The panoramic image processing system 200 of the present invention comprises an image capture module 210, an image processing device 220 and a correspondence generator 250. The correspondence generator 250 is optional.

[0054] The image capture module 210 is a camera with multiple lenses that can capture a field of view covering a horizontal FOV of up to 360 degrees and a vertical FOV of up to 180 degrees to generate multiple lens images. After receiving the lens images from the image capture module 210, the image processing device 220 generates a 2D color adjustment array with multiple color adjustment amounts for each lens image, and then performs a stitching operation based on the lens images and the 2D color adjustment array to form a panoramic image. Please note that the present invention does not limit the number of lenses included in the image capture module 210 as long as a FOV of up to 360 degrees horizontally and 180 degrees vertically can be captured. A necessary condition for setting up the image capture module 210 is that there should be enough overlap between the fields of view of any two adjacent lenses to facilitate image stitching. Examples of the panoramic image include, but are not limited to, a 360-degree panoramic image and an equidistant rectangular panoramic image.

[0055] For the sake of clarity and convenience of description, the following examples and embodiments are all described using equidistant rectangular panoramic images, and assume that the image capture module 210 includes four lenses, which are respectively mounted on four working surfaces (right, left, front, and back) of a cubic structure 11.

[0056] Figure 3A 2 is a flowchart of a method for averaging the color values ​​of all pixels in each feature region (executed by the image processing device 220) according to the first embodiment of the present invention. The method for averaging the color values ​​of all pixels in each feature region of the present invention is described below.

[0057] Step S302: Divide the area surrounding a texture / lens image j into a plurality of blocks. Figure 3BAn example showing a texture / lens image divided into eight blocks, each containing a feature area. Wherein j represents the index / ID of the texture / lens image and i represents the index / ID of the feature area in the texture / lens image j, wherein i=0-7.

[0058] Step S304: Determine the shape and size of each feature area and the texture coordinates of the feature points in each feature area. Figure 3B In the characteristic region The shape (such as a circle) is only an example and is not a limitation of the present invention. A feature point CP with texture coordinates i (j) Assigned to a corresponding feature region Each characteristic area Corresponds to a block and is located in the corresponding block. The feature point CP i (j) This feature area may be The shape, radius or size of the feature area of ​​the texture image j can be selected at will, as long as no two adjacent feature areas overlap, however, slight overlap does not hinder operation. The term "texture coordinates" refers to coordinates in a texture space (e.g., a texture / lens image j).

[0059] Step S306: Calculate the average color value of all pixels in each feature area of ​​the texture / lens image j (including brightness (Y; luma), chrominance (U, V; chroma) components). Please note that if a feature area is not located in Figure 1B For example, refer to Figure 1B and Figure 3B , characteristic area and There is no need to calculate their average color value.

[0060] Step S308: Determine whether all textures / lens images have been processed. If not, return to step S302 to process the next texture / lens image; if yes, end this process.

[0061] Figure 2 The processing pipeline of the panoramic image processing system 200 with the correspondence generator 250 of the present invention is also shown. The processing pipeline is divided into an offline phase and an online phase. In the online phase, the panoramic image processing system 200 generates a panoramic image. In the offline phase, the correspondence generator 250 calibrates the four lenses respectively, for example, Figure 1AMany circles are drawn on the surface of a sphere 12 with a radius of 2 meters (r=2) as longitudes and latitudes (not shown), and their multiple intersections are regarded as multiple calibration points; the four lenses of the image capture module 210 capture the calibration points, and the positions of the calibration points on the lens images are known. Then, because the view angles of the calibration points and the lens coordinates are linked, a mapping relationship between the equirectangular panoramic image and the lens image can be established. In this specification and the subsequent claims, a calibration point with the above mapping relationship is defined as a "vertex". After calibration, the correspondence generator 250 generates a correspondence table, which includes multiple vertices, and each vertex provides a mapping relationship between the equirectangular coordinates (latitude and longitude) and its texture coordinates in all lens images. In another embodiment, according to the correspondence table, the correspondence generator 250 uses a suitable image registration technology to generate an original vertex list (such as Table 1), and each vertex in the original vertex list provides a vertex mapping relationship between the equirectangular panoramic image and the lens image (or between the equirectangular coordinates and the texture coordinates). In the offline stage, the correspondence generator 250 will complete all necessary pre-calculations.

[0062] Figure 4A Displays a triangular mesh modeling the surface of a sphere. Figure 4A , a triangular mesh is used to model the surface of a sphere 12. Figure 4B Displays a polygonal mesh that composes / models the equirectangular panoramic image. Figure 4A The triangular mesh is projected into an equirectangular shape. Figure 4B The polygonal mesh Figure 4B The polygon mesh is a collection of quadrilaterals and triangles. Figure 4A It is obtained by projecting multiple pole triangles Figure 4B The top row and the bottom row of the polygonal mesh, so only Figure 4B The top and bottom rows of the polygonal mesh are formed by a plurality of quadrilaterals. The term "vertical triangle" refers to a triangle having a vertex that is the vertex (nadir or zenith) of a triangular mesh that is used to model the surface of a sphere.

[0063] In the offline stage, the correspondence generator 250 generates a polygon mesh ( Figure 4B), calculates the isometric rectangular coordinates and texture coordinates of each vertex of the polygon mesh, and determines whether each vertex is a pole to generate the original vertex list. Finally, the correspondence generator 250 transmits the original vertex list to the image processing device 220. Once the original vertex list is generated, it is reused by the image processing device 220 for subsequent operations, such as splicing subsequent lens images. The original vertex list is a list of multiple vertices that form the polygon mesh ( Figure 4B ), and each vertex is defined by a corresponding data structure. The data structure defines a vertex mapping relationship between a destination space and a texture space (or between the isometric rectangular coordinates and the texture coordinates). In one embodiment, the data structure includes, but is not limited to, isometric rectangular coordinates, a vertex flag, the number of lens images covered / overlapped, the texture coordinates in each lens image, the ID of each lens image, and the splicing blending weight of each lens image.

[0064] Table 1 shows an example of the data structure of each vertex in the original vertex list.

[0065]

[0066] Figure 5A 2 is a flowchart of a method for averaging the color values ​​of all pixels in each feature region (executed by the image processing device 220) according to the second embodiment of the present invention.

[0067] Step S502: Determine the feature points of a plurality of feature areas in a texture / lens image j and the preset longitude and latitude coordinate values ​​(ie, equidistant rectangular coordinates) of the feature points. Figure 1A and Figure 5B . Figure 5B show Figure 1A The cube structure 11 has a plurality of feature points with predetermined longitude and latitude coordinates (i.e., equidistant rectangular coordinates). Please note that all feature points must be located in the overlapping area of ​​the texture / lens image (e.g., Figure 6B In this embodiment, each of the four texture / lens images has eight feature points CP i (j) , where i = 0 to 7 and j = 0 to 3. However, Figure 5B The preset longitude and latitude coordinate values ​​shown are only examples and are not limitations of the present invention. In actual implementation, other points can be selected as feature points as long as the points fall within the overlapping area.

[0068] Step S504: Determine the feature points CP in the texture / lens image j according to the correspondence table (or the original vertex list) and the longitude and latitude coordinates (i.e., equidistant rectangular coordinates). i (j) , where i=0-5. As described above, the data structure of each vertex in the original vertex list provides a vertex mapping relationship between the isometric rectangular coordinates and the texture coordinates; in addition, the correspondence table provides a mapping relationship between each vertex in the isometric rectangular coordinates and the texture coordinates. For example, by searching the original vertex list generated by the correspondence generator 250 using isometric rectangular coordinates of longitude 45 degrees and latitude 0 degrees, a matching vertex can be quickly found; then, because the above-mentioned vertex with longitude 45 degrees and latitude 0 degrees must be located in an overlapping area, referring to Table 1, the first texture coordinate (u1, v1) in the first texture image and the second texture coordinate (u2, v2) in the second texture image can be extracted from the data structure of the matching vertex.

[0069] Step S506: Determine the characteristic points of each characteristic area. As mentioned above, any shape and size is applicable to the feature area, as long as any two adjacent feature areas do not overlap, however, slight overlap does not hinder the operation. As mentioned above, because the upper and lower edges of each texture / lens image do not overlap with any texture / lens image and each feature point CP has been determined i (j) The texture coordinates of the upper and lower feature areas do not need to be determined. and The shape and size of the feature area and The average color value.

[0070] Step S508: Calculate the feature areas in the texture / lens image j The average color value of all pixels in (including Y, U, and V components), where i = 0 to 5.

[0071] Step S510: Determine whether all textures / lens images have been processed. If not, return to step S502 to process the next texture / lens image; if yes, end this process.

[0072] In another embodiment, the correspondence generator 250 provides a modified vertex list to the image processing device 220. The modified vertex list is a list of multiple vertices that form the polygon mesh ( Figure 4B ), and each vertex is defined by a corresponding data structure. Table 2 shows an example of the data structure of each vertex in the modified vertex list.

[0073]

[0074]

[0075] As shown in Table 2, compared with the data structure of each vertex in the original vertex list, in the data structure of each vertex in the modified vertex list, each lens image further includes the following five fields: warping coefficient index (idx k0 ,idx k1 ), the blending weight of the joining coefficient (Alpha k ), color value accumulator index (IA k ), color adjustment index (ide k0 ,ide k1 ) and color adjustment blending weight (cw k ), where k=1~N.

[0076] Please also refer to Figure 1A and Fig. 6A . Under an ideal condition, the four lenses (including lenses #1 and #2) of the image capture module 210 are simultaneously arranged at the system center of a cubic structure 11, so that a single ideal imaging point of a distant object is located on an imaging plane 12 with a radius of 2 meters (r=2). Accordingly, because the ideal imaging position of lens image #1 (from lens #1) and the ideal imaging position of lens image #2 (from lens #2) match, after the image processing device 220 completes the image stitching / blending operation, the equidistant rectangular panoramic image will present a perfect stitching / blending result. However, under actual conditions, the actual focal points (or camera centers) 36 and 37 of lenses #1 and #2 are separated from the optimal focus (or system center) 32, so the two actual imaging positions 38 of a distant object 35 are separated on the imaging plane 22, such as Fig. 6A For results, refer to Fig. 6A On the right side of the image processing device 220, after the image stitching / blending operation is completed, the equidistant rectangular panoramic image will clearly show mismatched image defects. Therefore, the texture coordinates of each vertex must be corrected to reduce the mismatched image defects caused by the offset lens center of the image capture module 210.

[0077] Figure 6B This is an example of displaying an equidistant rectangular panoramic image with twelve overlapping areas according to the prior art. Figure 6C This is an example of displaying an equidistant rectangular panoramic image with twenty control areas according to the prior art. Figure 6B to Figure 6CFor example, the twelve overlapping areas R(1)~R(12) are divided into twenty control areas B(1)~B(20), and the twenty control areas B(1)~B(20) respectively have twenty joining coefficients Cw(1)~Cw(20) or twenty joining degrees (warping degrees) Cw(1)~Cw(20).

[0078] by Fig.6D Take the target vertex P2 (with original texture coordinates (u2, v2) in the front lens image) as an example, assuming that a rotation angle θ is in the clockwise direction and θ=93°, because the front lens image has eight control regions, 360° / 8=45°, idx=θ / 45°=2, and θmod 45°=θ-idx×45°=3°. Therefore, the two control regions closest to the vertex P2 are B(18) and B(11), and the interpolation joint coefficient / degree C' of the target vertex P2 in the front lens image is defined / calculated as follows: C'=Cw(18)×(3 / 45)+Cw(11)*(1-3 / 45). The angle θ is formed between a first vector V1 and a second vector V2; the first vector V1 starts from the image center O and ends at the position 62 of the starting control area B (7), while the second vector V2 starts from the image center O and ends at the target vertex P2. The position 62 of each control area includes, but is not limited to, the gravity center of the control area, the area center of the control area, and the median point of the control area.

[0079] To simplify the above calculation, taking the target vertex P2 as an example, in the offline stage, the correspondence generator 250 additionally pre-determines which two control regions (i.e., B(18) and B(11)) are the closest to the target vertex P2, and writes their index values ​​(18 and 11) into the "joining coefficient index" field of the front lens image of the data structure of the target vertex P2 in the modified vertex list (refer to Table 2); in addition, the correspondence generator 250 additionally pre-calculates the blending weight (=3 / 45) of the joining coefficients (Cw(18) and Cw(11)), and writes the blending weight into the "joining coefficient blending weight (Alpha)" field of the front lens image of the data structure of the target vertex P2 in the modified vertex list. Please note that the joining coefficients (Cw(1)-Cw(20)) are stored and arranged as a one-dimensional joining coefficient array Cw or a 1D data stream.

[0080] Please refer to Figure 7The image processing device 220 includes a computing unit 710 and a rendering engine 750. The rendering engine 750 includes a rasterization engine 75, a texture mapping circuit 72, a blending unit 73, and a destination buffer 74. The term "rasterization" refers to the computational process of mapping scene geometry (or a panoramic image) to texture coordinates of each lens image. After receiving the modified vertex list, the computing unit 710 extracts two joint coefficients (Cw(18) and Cw(11)) from a one-dimensional joint coefficient array Cw according to the content of the "joint coefficient index" field of the front lens image in the data structure of the target vertex P2 (refer to Table 2) (i.e., 18 and 11), and calculates the interpolated joint coefficient / degree C' of the target vertex P2 according to the "joint coefficient blending weight (Alpha)" field of the front lens image in the data structure of the target vertex P2 (i.e., 3 / 45). Next, the calculation unit 710 uses the following equation: u2′=(u2-u center )×C'+u center ; v2' = (v2-v center )×C'+v center , calculate the modified texture coordinates (u2', v2') of the target vertex P2 (with original texture coordinates (u2, v2)) in the front lens image, where the image center O of the front lens image has texture coordinates (u center ,v center ). Finally, the computing unit 710 updates the texture coordinates in the front lens image in the data structure of the target vertex P2 with the modified texture coordinates (u2', v2'). In this way, the computing unit 710 sequentially modifies all the texture coordinates in the data structure of all the vertices in the modified vertex list according to the one-dimensional joint coefficient array Cw, so as to transmit three vertices with the modified texture coordinates to the display engine 750 each time. After modifying all the texture coordinates in the data structure of all the vertices in the modified vertex list and completing the above-mentioned stitching process, the mismatched image defect caused by the offset lens center can be greatly improved. As for the determination process of the twenty joint coefficients Cw(1)~Cw(20) of the above-mentioned twenty control areas B(1)~B(20) (arranged into the one-dimensional joint coefficient array Cw), and the detailed process of the image processing device 220 for stitching, they have been disclosed in the U.S. Patent No. 10,810,700B2 (the content of the above-mentioned patent is hereby cited as a part of the content of this specification), and will not be repeated here.

[0081] In one embodiment, the correspondence generator 250 pre-sets Figure 1BEach overlapping area 13 is divided into three feature areas, and each feature area can be identified according to its equidistant rectangular coordinates. According to the three vertices with modified texture coordinates from the calculation unit 710, the display engine 750 performs triangle rasterization operation, texture mapping operation and blending operation, and prompts the measurement unit 77 to calculate the average color value of each feature area. Specifically, after receiving the three vertices with modified texture coordinates, a rasterization engine 75 determines the size of the number N of images covered / overlapped. Since the image capture module 210 is a camera with four lenses, N<=2. If N=1, the rasterization engine 75 sets a control signal CS1 to invalid (de-assert) to disable the measurement unit 77; otherwise, the rasterization engine 75 sets the control signal CS1 to valid (assert) to enable the measurement unit 77. For a point / pixel Q (with isometric rectangular coordinates (x, y)) located in a triangle (formed by the three vertices A, B, C with modified texture coordinates), the rasterization engine 75 calculates N pairs of texture coordinates and N working surface blending weights (fw1-fw2) in N lens images. N ). Taking N=2 as an example, the rasterization engine 75 uses a barycentric weighting method to calculate the equidistant rectangular coordinates (x A ,y A ,x B ,y B ,x C ,y C ,x D ,y D ,x,y), calculate three spatial weights (a,b,c). Based on the three spatial weights (a,b,c) and the texture coordinates of the three vertices (A, B, C), the rasterization engine 75 calculates two pairs of texture coordinates of the pixel Q as follows: 1Q ,v 1Q )=(a×u 1A +b×u 1B +c×u 1C ,a×v 1A +b×v 1B +c×v 1C );(u 2Q ,v 2Q )=(a×u 2A +b×u 2B +c×u 2C ,a×v 2A +b×v 2B +c×v 2C ), where (u 1Q ,v 1Q ),(u 1A,v 1A )(u 1B ,v 1B ) and (u 1C ,v 1C ) represents the texture coordinates of the pixel Q and the three vertices (A, B, C) in the first lens image, and (u 2Q ,v 2Q ),(u 2A ,v 2A )(u 2B ,v 2B ) and (u 2C ,v 2C ) represents the texture coordinates of the pixel Q and the three vertices (A, B, C) in the second lens image. According to the two pairs of texture coordinates of the pixel Q, the two texture mapping engines 721-722 of the texture mapping circuit 72 use any suitable method (such as nearest-neighbor interpolation, bilinear interpolation, or trilinear interpolation) to texture map the texture data of the above two lens images to generate two sample values ​​(s1-s2) of the pixel Q in parallel. Each sample value (s1-s2) can be a luma value or a chroma value. Then, according to the data structure of each of the three vertices (A, B, C) from the calculation unit 710, the rasterization engine 75 transmits the color value accumulator indexes (IA1-IA2) of the first and second lens images to the measurement unit 77.

[0082] For the above point / pixel Q, the measuring unit 77 uses a color value accumulator (IA1) (not shown) to accumulate its sample value s1, and uses a pixel counter (IA1) (not shown) to calculate the number of its pixels (whose sample values ​​are accumulated by the color value accumulator (IA1)); the measuring unit 77 uses a color value accumulator (IA2) (not shown) to accumulate its sample value s2, and uses a pixel counter (IA2) (not shown) to calculate the number of its pixels (whose sample values ​​are accumulated by the color value accumulator (IA2)). In this way, the measuring unit 77 accumulates the sample values ​​of each of the twenty-four feature regions. The sample values ​​and pixel numbers of all points / pixels in the image are calculated. Because the image capture module 210 includes four lenses (right, left, front, and back) and only six of the eight feature regions of each lens image need to calculate the average color value, there are a total of 24 feature regions (need to calculate the average color value), 24 color value accumulators, and 24 pixel counters. Finally, after all vertices related to the modified vertex list are transmitted to the rasterization engine 75, the measurement unit 77 calculates the average color value of each feature region according to the output values ​​of the corresponding color value accumulator and the corresponding pixel counter. The average color value (including Y, U, V components).

[0083] According to the equidistant rectangular coordinates of the vertices, it is easy to identify whether a vertex is located in / belongs to one of the feature areas. The following three methods can determine the texture coordinates of the feature points of each feature area. (1) Before correction and vertex level: For each feature area, the calculation unit 710 respectively accumulates the total number of vertices and the original texture coordinates of the vertices from the corrected vertex list to calculate the average original texture coordinates of the vertices as the texture coordinates of the feature points of each feature area. (2) After correction and vertex level: For each feature area, the calculation unit 710 respectively accumulates the total number of vertices and the corrected texture coordinates of the vertices related to the corrected vertex list to calculate the average corrected texture coordinates of the vertices as the texture coordinates of the feature points of each feature area. (3) After correction and pixel level: The measurement unit 77 receives the texture coordinates of all pixels (not shown), and respectively accumulates the total number of pixels and the texture coordinates of the pixels of each feature area to calculate the average texture coordinates of the pixels as the texture coordinates of the feature points of each feature area.

[0084] One of the features of the present invention is to regard two feature points located in two feature areas of two adjacent textures / lens images as a working pair, and determine the color adjustment amount of the two feature points according to the color difference (including brightness difference and chromaticity difference) between the two feature points of the working pair. Fig. 8A In the example, two adjacent feature areas of the texture image of two adjacent working surfaces 0 and 1 (such as ) of two characteristic points (such as ) is considered a working pair.

[0085] Figure 8B 2 is a flowchart showing a method for compensating color differences between adjacent lens images in a panoramic image (executed by the image processing device 220) according to a first embodiment of the present invention. The following describes a method for compensating color differences between adjacent lens images in a panoramic image according to the present invention.

[0086] Step S802: Calculate each feature area The average color value And determine the characteristic points CP of all characteristic areas i (j) Texture coordinates (or radii relative to the image center), where i = 0 to 5 and j = 0 to 3. Figure 3A and Figure 5A Methods and Figure 7 Related instructions.

[0087] Step S804: Calculate the color difference (including Y, U, and V components) between two feature points in each working pair. For example, calculate the feature points The color difference is as follows: And calculate the feature points The color difference is as follows:

[0088] Step S806: Determine all feature points CP i (j) The initial color adjustment amount Where i = 0 to 5 and j = 0 to 3. For example, two feature points are The initial color adjustment amount is set to and The initial color adjustment amount of a working pair usually corresponds to its color difference, for example, its color difference divided by an integer. Furthermore, the sum of the initial color adjustment amounts of two feature points in a working pair is usually less than the absolute value of their color difference. Please note that the above initial color adjustment amount is only an example and not a limitation of the present invention. In actual implementation, other values ​​are also applicable to the above initial color adjustment amount. Please also note that the two feature points The initial color adjustment amount of the pixel (which will not form any working pair with any feature point of its adjacent texture image) is set to zero.

[0089] Step S808: Using a low-pass filter to correct the initial color adjustment amount To obtain the final color adjustment value of all feature areas in a texture image j Where i = 0 to 7 and j = 0 to 3. For example, Figure 8C In FIG. 8 , the eight initial color adjustment values ​​(including Y, U, and V components) of the eight feature regions on the left are low-pass filtered by the low-pass filter on the right to obtain the eight final color adjustment values ​​(including Y, U, and V components) below. Please note that the order of steps S806 and S808 is interchangeable.

[0090] Step S809: Calculate the color adjustment amount (including Y, U, V components) of each vertex in a texture image j according to the distance / radius between each vertex and its two adjacent feature points and the final color adjustment amount of the two adjacent feature points. Fig.8D An example showing the positional relationship between the image center O, a target vertex S, and two adjacent feature points (G and H) in a frontal lens image. Fig.8D , from the first side and the second side Forming the first angle ∠GOS and from the third side and the second side The second angle ∠HOS is formed, where X is an interpolation point and the two vectors and have the same direction. Assume that the feature point G has a final color adjustment amount The distance / radius from the image center O of the front lens image (ie, texture image j) is RG, and the feature point H has a final color adjustment amount The distance / radius relative to the image center O is RH and a blending weight α=∠GOS∠HOS. Then, according to the following equations (1) and (2), the color adjustment amount of the interpolation point X can be calculated: And the distance / radius RX is as follows: RX=(1-α)×RG+α×RH……(2). Finally, according to equation (3), the color adjustment amount of the target vertex S is calculated. (including Y, U, V components) as follows: in, represents the distance / radius between the image center O and the target vertex S. In this way, the color adjustment amount of all vertices in the texture image j can be obtained, where j=0-3.

[0091] Step S810: In the lens image j, according to the color adjustment amount and texture coordinates of multiple vertices (such as three / four vertices, which can form a polygon) and the texture coordinates of each pixel, calculate the polygon (such as Figure 4B For example, if three vertices with color adjustment amounts and texture coordinates form a triangle and a target pixel S0 is located in the triangle, a color adjustment plane equation (such as f(u,v)=p×u+q×v+r) is used to obtain the color adjustment amount of the target pixel S0 according to the target pixel S0 and the texture coordinates of the three vertices; for example, the coefficient values ​​(p, q, r) of the color adjustment plane equation (f(u,v)=p×u+q×v+r) in the texture image j can be determined according to the color adjustment amounts and texture coordinates of the three vertices, and then, according to the texture coordinates (u 0S ,v 0S ) and the color adjustment plane equation, the color adjustment amount of the target pixel S0 located in the triangle (composed of the three vertices) of the texture image j is calculated as follows: ΔC0S =p×u 0S +q×v 0S +r. If four vertices with color adjustment values ​​and texture coordinates form a quadrilateral and a target pixel S1 is located within the quadrilateral, according to the texture coordinates (u 1S ,v 1S ), a bilinear interpolation method is used to obtain the color adjustment value of the target pixel S1. In this way, the color adjustment values ​​of all pixels in the texture image j can be obtained. The color adjustment values ​​of all pixels in the texture image j are stored in a separate two-dimensional color adjustment array.

[0092] Step S811: Calculate the color value (including Y, U, V components) of each pixel in the texture image j according to the sample value (including Y, U, V components) of each pixel in the texture image j and the color adjustment amount (including Y, U, V components) in the corresponding two-dimensional color adjustment array. When this step is completed, a color-compensated texture image corresponding to the texture image j can be formed.

[0093] Step S812: Determine whether all texture / lens images have been processed. If not, return to step S808 to process the next texture / lens image; if yes, end this process and four color-compensated lens / texture images have been formed.

[0094] Fig. 8E 2 is a flow chart of a method (executed by the image processing device 220) for compensating color differences between adjacent lens images in a panoramic image according to a second embodiment of the present invention. Fig. 8E In the method, since steps S802 to S808 and steps S811 to S812 are Figure 8B The same, so I will not repeat it here.

[0095] Step S818: Calculate the color adjustment amount (including Y, U, V components) of each pixel in a texture / lens image j according to the distance / radius of each pixel and its two adjacent feature points relative to the image center and the final color adjustment amount of the two adjacent feature points. Fig.8D In this embodiment, the target point S is regarded as a pixel. Similar to the calculation method described in step S809, the color adjustment amount of the target pixel S in the texture image j is calculated according to the above equations (1) to (3). (including Y, U, and V components). In this way, the color adjustment values ​​of all pixels in the texture image j can be obtained, where j=0-3.

[0096] Please note that before the image processing device 220 performs normal rasterization, texture mapping and blending operations on the four color-compensated lens / texture images (output of step S812), the above-mentioned Figure 8B and Fig. 8E The compensation method is Figure 8B and Fig. 8E The compensation method can be regarded as a pre-processing operation.

[0097] Fig.9A 2 is a flowchart of a method (executed by the image processing device 220) for obtaining the color adjustment amount and radius of multiple feature points in all lens images in a panoramic image according to an embodiment of the present invention. Fig.9A In the method, since step S802 and step S804 are Figure 8B The same, so I will not repeat it here.

[0098] Step S920: Correcting multiple feature points in a lens image j using a low-pass filter The color difference (e.g. ) to obtain the feature points The color difference after filtering (including Y, U, V components). Before filtering / correction, the two feature points (which will not form any working pair with any feature point in its neighboring texture image) ) is set to 0, where j=0~3.

[0099] Step S922: Based on the feature points of the lens image j The color difference after filtering determines the feature point Please note that the order of step S920 and step S922 is interchangeable.

[0100] Step S924: Determine whether all textures / lens images have been processed. If not, return to step S920 to process the next texture / lens image; if yes, end this process. At the end of this step, 32 feature points 32 color adjustment levels and 32 distances / radii (relative to the image center of the image itself) are stored respectively. Specifically, the 32 color adjustment amounts The 32 distances / radii are stored and arranged as a one-dimensional color adjustment amount array CA[m], and the 32 distances / radii are stored and arranged as a one-dimensional radius array RA[m], where m=0-31, j=0-3, and i=0-7. Fig.9A The purpose of the method is to help Figure 9B to Figure 9CThe one-dimensional color adjustment amount array CA[m] and the one-dimensional radius array RA[m] are obtained or set by a method.

[0101] In the offline stage, the correspondence generator 250 additionally predetermines which two feature points are closest to a target vertex (eg Fig.8D The two feature points G and H are closest to a target vertex S), and the index values ​​of the two feature points (idx k0 and idx k1 ; For example: CA[ide k0 ]=RH and CA[ide k1 ]=RG) is written into the "color adjustment index" field of the texture image j in the data structure of the target vertex in the modified vertex list (refer to Table 2); and the correspondence generator 250 additionally pre-calculates the color adjustment blending weights (for example: Fig.8D Medium k =∠GOS / ∠HOS=α), and writes the color adjustment blending weight into the “color adjustment blending weight” field of the texture image j in the data structure of the target vertex in the modified vertex list. In the same way, in the offline stage, the correspondence generator 250 calculates and fills the “color adjustment blending weight” field and the “color adjustment index” field of each texture image in the data structure of all vertices in the modified vertex list.

[0102] FIG. 9B to FIG. 9C FIG. 2 is a flow chart of a method (executed by the image processing device 220) for compensating color differences between adjacent lens images in a panoramic image according to a third embodiment of the present invention. FIG. 9B to FIG. 9C The purpose of the method is to apply the one-dimensional color adjustment amount array CA[m] and the one-dimensional radius array RA[m] to multiple input lens images to form a color-compensated panoramic image. Figure 7 and FIG. 9B to FIG. 9C , a method for compensating color differences between adjacent lens images in a panoramic image is described.

[0103] Step S962: Take a vertex from a modified vertex list (refer to Table 2).

[0104] Step S964: Calculate the color adjustment amount of the vertex in each lens image according to the texture coordinates of each lens image in the data structure of the vertex and the values ​​in the "color adjustment blending weight" field and the "color adjustment index" field of each lens image. Assume that the vertex C from the modified vertex list is covered / overlapped by two lens images (front face, top face; N=2), and the vertex C has the following data structure and its texture coordinates have been modified by the computing unit 710: vertex C: {(x C ,y C ),2,0,IDFront ,(u 1C ,v 1C ),w 1C ,(idx 10C ,idx 11C ),Alpha 1C ,IA 1C ,(ide 10C ,ide 11C ),cw 1C ,ID Top ,(u 2C ,v 2C ),w 2C ,(idx 20C ,idx 21C ),Alpha 2C ,IA 2C ,(ide 20C ,ide 21C ),cw 2C}. According to the formula: d 1C =sqrt((u 1C -u O )×(u 1C -u O )+(v 1C -v O )×(v 1C -v O )), the rasterization engine 75 calculates the distance d between the vertex C in the front lens image and the image center O of the front lens image. 1C Afterwards, the rasterization engine 75 takes out a weight cw from the "color adjustment blending weight" field of the front lens image in the data structure of the vertex C. 1C , and take out a pair of index values ​​(ide 10C ,ide 11C ). Then, according to the formula: R 1XC =cw 1C ×RA[ide 10C ]+(1-cw 1C )×RA[ide 11C ], the rasterization engine 75 calculates the radius R of an interpolated vertex Xc in the front lens image 1XC , then according to the formula: ΔC 1C =(cw 1C ×CA[ide 10C ]+(1-cw 1C )×CA[ide 11C ])×(d 1C / R 1XC), calculate the color adjustment amount ΔC of the vertex C in the front lens image 1C In the same way, according to the formula: ΔC 2C =(cw 2C ×CA[ide 20C ]+(1-cw 2C )×CA[ide 21C ])×(d 2C / R 2XC ), the rasterization engine 75 can calculate the color adjustment amount ΔC of the vertex C in the top lens image. 2C .

[0105] Step S966: Check whether the vertex and the two vertices in front of it (taken from the modified vertex list) form a triangle. If not, jump to step S976; if yes, jump to step S968.

[0106] Step S968: Calculate the color adjustment amount of each pixel in the triangle in each lens image according to the color adjustment amounts of the three vertices and the texture coordinates of each pixel in the triangle.

[0107] Assume that vertex C and its two preceding vertices (A, B) from the modified vertex list are covered / overlapped by two lens images (front and top; N=2), and the two vertices (A, B) have the following data structures and their texture coordinates have been modified by the computing unit 710: vertex A: {(x A ,y A ),2,0,ID Front ,(u 1A ,v 1A ),w 1A ,(idx 10A ,idx 11A ),Alpha 1A ,IA 1A ,(ide 10A ,ide 11A ),cw 1A ,ID Top ,(u 2A ,v 2A ),w 2A ,(idx 20A ,idx 21A ),Alpha 2A ,IA 2A ,(ide 20A ,ide 21A ),cw 2A};vertex B:{(x B ,y B ),2,0,ID Front,(u 1B ,v 1B ),w 1B ,(idx 10B ,idx 11B ),Alpha 1B ,IA 1B ,(ide 10B ,ide 11B ),cw 1B ,ID Top ,(u 2B ,v 2B ),w 2B ,(idx 20B ,idx 21B ),Alpha 2B ,IA 2B ,(ide 20B ,ide 21B ),cw 2B}. Similar to the calculation method described in step S964, according to the formula: ΔC 1A =(cw 1A ×CA[ide 10A ]+(1-cw 1A )×CA[ide 11A ])×(d 1A / R 1XA ) and ΔC 1B =(cw 1B ×CA[ide 10B ]+(1-cw 1B )×CA[ide 11B ])×(d 1B / R 1XB ), the rasterization engine 75 calculates the color adjustment amount (ΔC 1A , ΔC 1B ); In addition, according to the formula: ΔC 2A =(cw 2A ×CA[ide 20A ]+(1-cw 2A )×CA[ide 21A ])×(d 2A / R 2XA );ΔC 2B =(cw 2B ×CA[ide 20B ]+(1-cw 2B )×CA[ide 21B ])×(d 2B / R 2XB), the rasterization engine 75 calculates the color adjustment amount (ΔC) of the two vertices (A, B) in the top lens image. 2A , ΔC 2B Then, according to the color adjustment amount (ΔC 1A , ΔC 1B , ΔC 1C ) and texture coordinates, the rasterization engine 75 determines the three parameter values ​​(p, q, r) in a color adjustment plane equation (f(u, v) = p×u+q×v+r) in the front lens image. Then, according to the color adjustment plane equation (ΔC 1T =p×u 1T +q×v 1T +r), the rasterization engine 75 calculates a target pixel T (u ) located in a triangle defined by the three vertices (A, B, C) in the front lens image. 1T ,v 1T ) color adjustment amount ΔC 1T In the same manner, the rasterization engine 75 calculates the target pixel T (u 2T ,v 2T ) color adjustment amount ΔC 2T In addition, the rasterization engine 75 calculates two working surface blending weights (fw1-fw2) of the front and top lens images respectively according to the three spatial weights (a, b, c) of the three vertices and the three splicing blending weights (please refer to Table 2) as follows: fw1 = a×w 1A +b×w 1B +c×w 1C ; fw2=a×w 1A +b×w 1B +c×w 1C Afterwards, the rasterization engine 75 transmits the two working plane blending weights (fw1-fw2) and two color adjustment values ​​(ΔC 1T , ΔC 2T ) to the blending unit 73. In the same manner, the rasterization engine 75 calculates the color adjustment amount of each pixel in the triangle (A, B, C) in each lens image.

[0108] Step S970: Calculate the texture coordinates of each pixel in the triangle in each lens image based on the texture coordinates of the three vertices in each lens image. Taking the target pixel T and the three vertices (A, B, C) in step S968 as an example, the texture coordinates of the target pixel T in the front and top lens images are calculated as follows: 1T ,v 1T )=(a×u1A +b×u 1B +c×u 1C ,a×v 1A +b×v 1B +c×v 1C );(u 2T ,v 2T )=(a×u 2A +b×u 2B +c×u 2C ,a×v 2A +b×v 2B +c×v 2C ). Among them, (u 1A ,v 1A )、(u 1B ,v 1B ) and (u 1C ,v 1C ) represents the texture coordinates of the three vertices (A, B, C) in the front lens image, (u 2A ,v 2A )、(u 2B ,v 2B ) and (u 2C ,v 2C ) represents the texture coordinates of the three vertices (A, B, C) in the top lens image and (a, b, c) represent the three spatial weights.

[0109] Step S972: Texture-map the texture data of the lens image according to the texture coordinates of each pixel in the triangle. Taking the target pixel T and the three vertices (A, B, C) in step S970 as an example, according to the two texture coordinates (u 2T ,v 2T ) and (u 2T ,v 2T ), the two texture mapping engines 721-722 utilize any suitable method (such as nearest neighbor interpolation, bilinear interpolation, or trilinear interpolation) to texture map the texture data of the front and top lens images to generate two sample values ​​s1 and s2.

[0110] Step S974: Calculate the color value (including Y, U, V components) of each pixel in the triangle according to the sampled value, color adjustment amount and working plane blending weight of each pixel in the triangle. Taking the target pixel T and three vertices (A, B, C) in step S972 as an example, the blending unit 73 receives two working plane blending weights (fw1-fw2) and two color adjustment amounts (ΔC 1T , ΔC 2T) and after receiving two sample values ​​s1 and s2 from the texture mapping engines 721-722, the following equation is used to mix the two sample values ​​s1 and s2 to generate the color value Vb of the pixel T: Vb = fw1×(s1+ΔC 1T )+fw2×(s2+ΔC 2T ), where fw1+fw2=1. Finally, the blending unit 73 stores the color value Vb of the pixel T in a corresponding position in the target buffer 74. In the same way, the color value of each pixel in the triangle (A, B, C) can be calculated.

[0111] Step S976: Check whether all vertices have been processed. If not, return to step S962; if yes, end this process and form a color-compensated panoramic image.

[0112] In another example, assuming that three vertices (A, B, C) from the modified vertex list are only covered by one lens image (front view; N=1), the calculation method is similar except that only one lens image needs to be processed. The blending unit 73 receives a working face blending weight (fw1), a color adjustment amount (ΔC 1T ) and a sample value s1, the color value Vb of the pixel T is generated using the following equation: Vb = fw1 × (s1 + ΔC 1T ), where fw1=1.

[0113] The image processing device 220 and the correspondence generator 250 of the present invention can be implemented by software, hardware, or a combination of software (or firmware) and hardware. An example of a simple solution is a field programmable gate array or an application specific integrated circuit. In a preferred embodiment, the image processing device 220 is implemented using a graphics processing unit (GPU) and a first program memory; the correspondence generator 250 is implemented using a general-purpose processor and a second program memory. The first program memory stores a first processor executable program, and the second program memory stores a second processor executable program. When the graphics processing unit executes the first processor executable program, the graphics processing unit is configured to operate as: the image processing device 220. When the general-purpose processor executes the second processor executable program, the general-purpose processor is configured to operate as: the correspondence generator 250.

[0114] In another embodiment, the image processing device 220 and the correspondence generator 250 are implemented using a general-purpose processor and a third program memory. The third program memory stores a third processor executable program. When the general-purpose processor executes the third processor executable program, the general-purpose processor is configured to operate as: the correspondence generator 250 and the image processing device 220.

[0115] The above are only preferred embodiments of the present invention and are not intended to limit the scope of the present invention; any other equivalent changes or modifications that do not deviate from the spirit disclosed by the present invention should be included in the scope of the following patent applications.

Claims

1. A method for compensating color differences between adjacent lens images in a panoramic image, for compensating color differences between adjacent lens images in a panoramic image, each lens image comprising a plurality of feature areas and each feature area having a feature point, characterized in that: The method comprises: Obtaining an average color value of the feature area; Calculating the color difference of each pair of feature areas between any two adjacent lens images from a plurality of lens images generated by a camera with multiple lenses according to the average color value of the feature areas; Determining the color adjustment amount of the feature point in each lens image according to the color difference of each pair of feature areas; and Calculating a color adjustment amount of a unit according to positions of a unit and two adjacent feature points thereof and color adjustment amounts of the two adjacent feature points; The obtaining step comprises: For each pixel in a triangle formed by three vertices from a first vertex list, performing a triangle rasterization operation and a texture mapping operation on the modified texture coordinates of the three vertices to generate N sample values ​​corresponding to N lens images, where N>1; For each pixel in the triangle, respectively accumulate the N sample values ​​using a plurality of color value accumulators, wherein the color value accumulators are indexed by the N accumulator index values ​​in the data structures of the three vertices; For each pixel in the triangle, accumulating the number of pixels in the N lens images respectively using a plurality of pixel counters, wherein the pixel counters are indexed by the N accumulator index values; Repeating the steps of performing triangle rasterization and texture mapping operations, respectively accumulating the N sample values, and respectively accumulating the number of pixels in the N lens images until all vertices from the first vertex list are processed; and Calculate the average color value of each feature area in each lens image according to the output value of a corresponding color value accumulator and a corresponding pixel counter; The first vertex list includes a plurality of vertices having a data structure, and the data structure defines a vertex mapping between the panoramic image and the lens image.

2. The method according to claim 1, characterized in that The multi-lens camera captures a field of view covering 360 degrees horizontally and up to 180 degrees vertically to produce the lens image.

3. The method according to claim 1, characterized in that Each feature point is the centroid of its corresponding feature area.

4. The method according to claim 1, characterized in that The obtaining step further comprises: Before the steps of performing triangle rasterization and texture mapping operations, the modified texture coordinates of the three vertices in each lens image are calculated according to the corresponding connection degree and original texture coordinates of the three vertices in each lens image.

5. The method according to claim 1, characterized in that Also includes: According to the total number of vertices in each feature area, the original texture coordinates of the vertices in each feature area are averaged respectively to determine the texture coordinates of each feature point in each feature area in each lens image.

6. The method according to claim 1, characterized in that Also includes: According to the total number of vertices in each feature area, the modified texture coordinates of the vertices in each feature area are averaged respectively to determine the texture coordinates of each feature point in each feature area in each lens image.

7. The method according to claim 1, characterized in that Also includes: According to the output value of the pixel counter, the texture coordinates of the pixels in each feature area are respectively averaged to determine the texture coordinates of each feature point in each feature area in each lens image.

8. The method according to claim 1, characterized in that The decision step includes: According to the color difference of each pair of feature areas, the initial color adjustment amount of all feature points is determined; Using a low-pass filter, correcting an initial color adjustment amount of the feature point in a lens image; and The correction step is repeated until all lens images are processed to obtain the color adjustment amount of the feature point in each lens image.

9. The method according to claim 1, characterized in that The decision step includes: Using a low-pass filter, correcting the color difference of the feature point in a lens image to generate a filtered color difference of the feature point; determining a color adjustment amount of the feature point in the lens image according to the filtered color difference value of the feature point; and The correction step and the step of determining the color adjustment amount of the feature point in the lens image are repeated until all lens images are processed to obtain the color adjustment amount of the feature point in each lens image.

10. The method according to claim 1, characterized in that Also includes: The color value of a pixel is calculated according to the sampling value and the color adjustment amount of the pixel.

11. The method according to claim 10, characterized in that Also includes: The step of calculating the color adjustment amount of the unit and the step of calculating the color value of the pixel are repeated until all pixels in a lens image are processed to form a lens image after color compensation.

12. The method according to claim 11, characterized in that The unit is a pixel.

13. The method according to claim 10, characterized in that The unit includes a plurality of vertices forming a polygon and the step of calculating the color adjustment amount of the unit includes: In a lens image, calculating a color adjustment amount of each of the plurality of vertices according to the radius of each of the plurality of vertices and the radius and color adjustment amount of two adjacent feature points; and In the lens image, a color adjustment amount of the pixel is calculated according to the texture coordinates of the pixel located in the polygon and the color adjustment amounts of the plurality of vertices.

14. The method according to claim 13, characterized in that The polygon is a quadrilateral and the step of calculating the color adjustment amount of the pixel comprises: In the lens image, a color adjustment amount of the pixel is calculated according to the color adjustment amounts and texture coordinates of four vertices forming a quadrilateral and the texture coordinates of the pixel located in the quadrilateral by using a bilinear interpolation method.

15. The method according to claim 13, characterized in that The polygon is a triangle and the step of calculating the color adjustment amount of the pixel comprises: In the lens image, determining a color adjustment plane equation according to the color adjustment amounts and texture coordinates of three vertices forming the triangle; and In the lens image, the color adjustment amount of the pixel is calculated according to the color adjustment plane equation and the texture coordinates of the pixel located in the triangle.

16. The method according to claim 10, characterized in that The unit includes three vertices forming a triangle and the step of calculating the color adjustment amount of the unit includes: (a) calculating the color adjustment amount of each of the three vertices in each of the N lens images according to N texture coordinates in the N lens images, N color adjustment blending weights, and N pairs of index values ​​corresponding to a first array and a second array in a data structure of each of the three vertices from a second vertex list, wherein N represents the number of overlapping lens images of the three vertices and N>1; (b) determining a plurality of parameter values ​​of a color adjustment plane equation according to the color adjustment amounts and texture coordinates of the three vertices in each of the N lens images; and (c) calculating the color adjustment amount of the pixel in each of the N lens images according to the color adjustment plane equation and the texture coordinates of the pixel located in the triangle; The color adjustment amount and radius of the feature area are stored in the first array and the second array respectively; and The second vertex list includes multiple vertices with a data structure, and the data structure defines the vertex mapping between the panoramic image and the lens image, and each data structure includes the texture coordinates of each lens image, the color adjustment blending weight and a pair of index values ​​corresponding to the first array and the second array.

17. The method according to claim 16, characterized in that The step of calculating the color value of the pixel comprises: (d) calculating N texture coordinates of the pixel located in the triangle in the N lens images according to the spatial weights of the three vertices and the texture coordinates in the N lens images; (e) texture mapping the texture data of the N lens images according to the N texture coordinates of the pixel to generate N sample values; (f) calculating N working surface blending weights according to the spatial weights of the three vertices and the splicing blending weights of the N lens images; and (g) calculating the color value of the pixel according to the N working plane blending weights, the N sampling values ​​and the N color adjustment amounts of the N lens images; The spatial weights are related to the equidistant rectangular coordinates of the three vertices and the pixel; and Wherein, each data structure in the second vertex list further includes a splicing and blending weight of each of the N lens images.

18. The method according to claim 17, characterized in that The step of calculating the color value of the pixel further comprises: (h) repeating steps (c) to (g) until all pixels within the triangle are processed; and (i) Repeat steps (a) to (h) until all vertices are processed to form a color-compensated panoramic image.

19. The method of claim 1, wherein: The step of calculating the color adjustment amount of the unit comprises: According to the following equation: RX=(1-α)×RG+α×RH, the radius RX of an interpolation point X between two feature points G and H adjacent to the unit S in the lens image j is calculated; According to the following equation: Calculate the color adjustment amount of the interpolation point X as well as According to the following equation: Calculate the color adjustment amount of the unit S in, represents the distance between the image center O of the lens image j and the unit S, where α = ∠GOS / ∠HOS, and Respectively represent the color adjustment amount of the two feature points G and H in the lens image j, and RG, RH and RX represent the radius of the points G, H and X relative to the image center O, respectively, and the two vectors and have the same direction.

20. A method for compensating color differences between adjacent lens images in a panoramic image, for compensating color differences between adjacent lens images in a panoramic image, each lens image comprising a plurality of feature areas and each feature area having a feature point, characterized in that: The method comprises: Obtaining an average color value of the feature area; Calculating the color difference of each pair of feature areas between any two adjacent lens images from a plurality of lens images generated by a camera with multiple lenses according to the average color value of the feature areas; Determining the color adjustment amount of the feature point in each lens image according to the color difference of each pair of feature areas; and Calculating a color adjustment amount of a unit according to positions of a unit and two adjacent feature points thereof and color adjustment amounts of the two adjacent feature points; The obtaining step comprises: By calibrating the camera with multiple lenses, one of a correspondence table and a first vertex list is obtained; Selecting the feature point with preset longitude and latitude coordinates; Determining the texture coordinates of each feature point in each lens image according to the preset longitude and latitude coordinates of each feature point and one of the correspondence table and the first vertex list; Determine the shape and size of the corresponding feature area of ​​each feature point; as well as Calculate the average color value of each feature area; Wherein, the first vertex list includes a plurality of first vertices having a data structure, and the data structure defines a vertex mapping between the panoramic image and the lens image; The correspondence table includes a plurality of second vertices, and each second vertex provides a mapping between the panoramic image and the lens image.

Images