Method for generating projection images by scaling adjustment and seam splicing

Abstract

The present invention provides an image processing method for receiving M lens images and generating a projection image, the method comprising: determining P optimal joint coefficients of P control areas in the projection image according to a 2D error table and the M lens images, wherein an image capture module is used to capture a field of view with an X-degree horizontal field of view and a Y-degree vertical field of view to generate the M lens images; generating M work surface images according to a first vertex list, the P optimal joint coefficients and the M lens images; determining N seams of N seam areas; and, according to the seams of each seam area, splicing two overlapping seam images of each seam area to generate a spliced ​​seam image. The 2D error table comprises a plurality of test joint coefficients and a plurality of corresponding accumulated pixel value errors in the P control areas.

Description

Technical Field

[0001] The present invention relates to image processing, and more particularly to a method for generating projection images by scaling and seam splicing. The present invention is helpful for generating projection images in an image processing system. Background Art

[0002] Figure 1A The relationship between a cubic structure 11 and a sphere 12 is shown. Figure 1A As shown, a conventional camera has four lenses and is respectively mounted on four working surfaces (right, left, front, and back) of a cubic structure 11. Any two adjacent surfaces of the four working surfaces are substantially orthogonal. For example, the four working surfaces are respectively oriented toward the longitudes of 0, 90, 180, and 270 of the virtual sphere 12 to simultaneously capture a field of view with a 360-degree horizontal field of view (FOV) and a 90-degree vertical FOV, and generate four lens images. A necessary condition for setting up the camera with four lenses is that the fields of view of any two adjacent lenses should have sufficient overlap to facilitate image stitching. Please refer to Figure 1B , the pixels in the regions A(1) to A(2) are formed by the overlap of two lens / texture images, while the pixels in the region 15 are from a single lens / texture image. Therefore, the overlap region 13 can be stitched to form an equirectangular panoramic image.

[0003] A perfect projection image processing system should make it impossible for viewers to tell that the video is captured by multiple lenses, so that viewers can experience a seamless experience where multiple lenses look like a single lens. Summary of the invention

[0004] In view of the above problems, one of the objectives of the present invention is to provide an image processing method for generating a projection image by scaling adjustment and seam splicing.

[0005] According to an embodiment of the present invention, an image processing method is provided for receiving M lens images and generating a projection image. The method comprises: determining P optimal joint coefficients of P control areas in the projection image according to a 2D error table and the M lens images, wherein an image capture module is used to capture a field of view with an X-degree horizontal field of view and a Y-degree vertical field of view to generate the M lens images; generating M working surface images according to a first vertex list, the P optimal joint coefficients and the M lens images; determining N seams of N seam areas; and, according to the seams of each seam area, splicing two overlapping seam images of each seam area to generate a spliced ​​seam image. Wherein, each working surface image includes z seam images and one main image, and each seam area is located between two adjacent main images in the projection image; the 2D error table includes multiple test joint coefficients and multiple corresponding accumulated pixel value errors in the P control areas; the P control areas are respectively located within the N seam areas, and the N seam areas are respectively located within the N overlapping areas of the M working surfaces; the first vertex list includes multiple first vertices with a data structure, and the data structure defines a first vertex mapping between a single lens image and the projection image; wherein, 1<=z<=2, M>=2, N>=1, P>=3, X<=360 and Y<180.

[0006] The above and other objects and advantages of the present invention are described in detail below with reference to the following drawings, detailed description of the embodiments and scope of claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1A The relationship between a cubic structure and a sphere.

[0008] Figure 1B It is an equidistant rectangular panoramic image, which is derived from the equidistant rectangular projection of four lens images (back, left, right, and front) of an image capture module 21 .

[0009] Figure 2 FIG. 4 is a block diagram showing a projection image processing system according to an embodiment of the present invention.

[0010] Figure 3A A triangular mesh used to model the surface of a sphere.

[0011] Figure 3B A polygonal mesh used to compose / model the equirectangular panoramic image.

[0012] Figure 3C An example of an equidistant rectangular panoramic image with four overlapping areas and twenty control areas.

[0013] Figure 4AFIG. 1 is an example of the relationship between the target vertex P and the ten control regions R( 6 ) to R( 15 ) in the front lens image.

[0014] Figure 4B It is shown how the mismatched image defect of an object is improved after the vertex processing device 210 modifies the texture coordinates of all vertices according to the joining coefficients.

[0015] Figure 5 FIG. 4 is a schematic diagram showing the image processing device according to an embodiment of the present invention.

[0016] Fig. 6A FIG. 4 is a flow chart showing a method for determining the optimal bonding coefficients of all control zones according to an embodiment of the present invention.

[0017] Figure 6B 1 is a flow chart showing a method for the splicing decision unit 230 to perform the coefficient decision operation of step S612 according to an embodiment of the present invention.

[0018] Figure 6C Among all the paths of the link measurement, the total link measurement value of the path 0-1-1 (solid line path) is the smallest.

[0019] Fig. 7A An example of an equidistant rectangular panoramic image including four main images and their seam areas.

[0020] Figure 7B for Fig. 7A An enlarged view of the seam area S(2).

[0021] Figure 7C FIG. 4 is a flow chart showing a method for determining a minimum error path (or a seam) in a seam region according to an embodiment of the present invention.

[0022] Fig.7D This is an example where a seam 78 is located in a seam area and the seam area is located between a left main image and a right main image.

[0023] Fig. 8A The figure is a flow chart showing an image processing method according to an embodiment of the present invention.

[0024] Figure 8B 1 is an example of a section of a seam area, and the seam area is related to a seam 78, a seam image A and a seam image B.

[0025] Figure 8C The size of a seam area and the sizes and different positions of the update windows 81 of three rows.

[0026] Fig.8D A seam area is located at the far left and far right of the equidistant rectangular panoramic image.

[0027] Figure Number:

[0028] 11 Cube Architecture

[0029] 12 spheres

[0030] 15 Non-overlapping areas

[0031] 21 Image Capture Module

[0032] 22 Compensation device

[0033] 23 Correspondence Generator

[0034] 40 Ideal imaging position

[0035] 41, 44 Imaging Center

[0036] 42 Location of Control Zone

[0037] 43 System Center of Image Capture Module 21

[0038] 45 Objects

[0039] 46, 47 Lens Center

[0040] 48 Actual imaging position

[0041] 55 Image Engine

[0042] 56 Seam Engine

[0043] 71 intersections / vertices

[0044] 72 Quadrilateral

[0045] 78 Seams

[0046] 81 Update Window

[0047] 82 separation point

[0048] 200 Projection Image Processing System

[0049] 210 Vertex processing device 230

[0050] 220 Image processing device

[0051] 230 splicing decision unit

[0052] 510 Destination Buffer

[0053] 520 Temporary Buffer

[0054] 560 Rasterization Engine

[0055] 570 Texture Mapping Circuit

[0056] 571, 572 Texture Mapping Engine

[0057] 580 measurement units DETAILED DESCRIPTION

[0058] The singular terms "a", "an" and "the" mentioned in the entire specification and claims include both singular and plural meanings unless otherwise specified in the specification. The same reference symbols are used for circuit elements with the same function throughout the specification.

[0059] Figure 2 FIG. 2 is a block diagram of a projection image processing system according to an embodiment of the present invention. The projection image processing system 200 of the present invention comprises an image capture module 21, a compensation device 22 and a correspondence generator 23. The compensation device 22 receives the original vertex list from the correspondence generator 23 and multiple lens images from the image capture module 21 to generate a projection image. Please note that the image capture module 21 is a camera with multiple lenses, which can be a panoramic camera or a wide angle camera. Therefore, the projection image can be a panoramic image or a wide angle image.

[0060] Many projection methods are applicable to the projection image processing system 200 of the present invention. The term "projection" refers to: flattening the surface of a sphere into a two-dimensional plane, such as a projection plane. The projection includes, but is not limited to, equirectangular projection, cylindrical projection, and modified cylindrical projection. Modified cylindrical projection includes, but is not limited to, Miller projection, Mercator projection, Lambert cylindrical equalarea projection, Pannini projection, etc. Accordingly, the above-mentioned projection image includes, but is not limited to, an equirectangular projection image, a cylindrical projection image, and a modified cylindrical projection image. Figure 1A to Figure 1B and FIG. 3A to FIG. 3C Related to equirectangular projection. As for the implementation methods of cylindrical projection and modified cylindrical projection, they are well known to those skilled in the art and will not be described in detail here. Please note that no matter which projection method is adopted by the projection image processing system 200 of the present invention, the above-mentioned correspondence generator 23 will correspondingly generate an original vertex list (such as Table 1), which defines the vertex mapping relationship between the lens image and the projection image. For the sake of clarity and convenience of description, the following examples and embodiments are only illustrated with equirectangular projection and equirectangular panoramic images. It should be noted that the operation mode and image processing method of the projection image processing system 200 of the present invention ( FIG. 6A to FIG. 6B , Figure 7C and Fig. 8A ) also applies to the wide-angle camera, cylindrical projection, and modified cylindrical projection mentioned above.

[0061] The image capture module 21 is a camera with multiple lenses, which can simultaneously capture a field of view covering a horizontal FOV of X degrees and a vertical FOV of Y degrees to generate multiple lens images, where X<=360, Y<180, such as 360×160 or 180×90. Figure 1A As shown, the image capture module 21 includes four lenses (not shown) and is respectively mounted on four working surfaces of a cubic structure 11 to simultaneously capture a field of view with a 360-degree horizontal field of view FOV and a 90-degree vertical field of view FOV to generate four lens images. Please note that the present invention does not limit the number of lenses included in the image capture module 21 as long as a field of view of X degrees horizontal FOV and Y degrees vertical FOV can be captured, where X<=360, Y<180. A necessary condition for setting up the image capture module 21 is that the fields of view of any two adjacent lenses should overlap sufficiently to facilitate image stitching.

[0062] The definitions of the terms used throughout the specification and in the subsequent claims are as follows, unless otherwise specified in the specification. The term "texture coordinate" refers to a coordinate in a texture space (such as a texture image or a lens image); the term "rasterization operation" refers to the calculation process of mapping scene geometry (or a projection image) to the texture coordinates of each lens image. The processing pipeline of the projection image processing system 200 is divided into an offline phase and an online phase. In the offline phase, the four lenses of the image capture module 21 are calibrated respectively, and the correspondence generator 23 uses a suitable image registration technology to generate an original vertex list, and each vertex in the original vertex list provides a mapping relationship between the equirectangular panoramic image and the lens image (or between the equirectangular coordinate and the texture coordinate). For example, a plurality of circles are drawn on the surface of a sphere 12 with a radius of 2 meters (r=2) as longitude and latitude, and a plurality of intersections thereof are regarded as a plurality of calibration points. The four lenses capture the calibration points, and the positions of the calibration points on the lens images are known. Then, because the view angle of the calibration points and the texture coordinates are linked, a mapping relationship between the equirectangular panoramic image and the lens image can be established. In this specification, a calibration point with the above mapping relationship is defined as a "vertex". In short, the correspondence generator 23 calibrates the relationship between the equirectangular panoramic image and the lens image for each vertex to obtain the original vertex list. In the offline stage, the correspondence generator 23 completes all necessary calculations.

[0063] Figure 3A Displays a triangular mesh modeling the surface of a sphere. Figure 3A , a triangular mesh is used to model the surface of a sphere 12. Figure 3B Displays a polygonal mesh that composes / models the equirectangular panoramic image. Figure 3A The triangular mesh is projected into an equirectangular shape. Figure 3B The polygonal mesh Figure 3B A polygon mesh is a collection of quadrilaterals and / or triangles.

[0064] In the offline stage, according to the geometric shapes of the equidistant rectangular panoramic image and the lens image, the correspondence generator 23 is a polygonal mesh ( Figure 3B) calculates the isometric rectangular coordinates and texture coordinates of each vertex of the polygon mesh ( ) to generate the original vertex list. Then, the correspondence generator 23 transmits the original vertex list to the vertex processing device 210. The original vertex list is a list of multiple vertices that form the polygon mesh ( Figure 3B ), and each vertex is defined by a corresponding data structure. The data structure defines a vertex mapping relationship between a destination space and a texture space (or between the isometric rectangular coordinates and the texture coordinates). Table 1 shows an example of the data structure of each vertex in the original vertex list.

[0065] Table 1: Example of the data structure of each vertex in the original vertex list

[0066]

[0067]

[0068] For the sake of clarity and convenience of description, the following examples and embodiments assume that the image capture module 21 includes four lens images and is respectively mounted on four working surfaces (right surface, left surface, front surface, and back surface) of the cubic structure 11 .

[0069] Figure 3C An example showing an equidistant rectangular panoramic image with four overlapping areas and twenty control areas R(1) to R(20). Please also refer to Figure 1B , each overlapping area A(1) to A(4) includes P1 control areas arranged in a column, where P1>=3. The following examples and embodiments are described by taking the example that each overlapping area of ​​an equidistant rectangular panoramic image includes 5 (P1=5) control areas. Figure 3C In the example, the equidistant rectangular panoramic image has twenty control areas R(1)~R(20), and the control areas R(1)~R(20) respectively have twenty warping coefficients C(1)~C(20), and the twenty warping coefficients respectively represent the different degrees of warping of the twenty control areas R(1)~R(20).

[0070] The compensation device 22 includes a vertex processing device 210, a stitching decision unit 230, and an image processing device 220. In a measurement mode, the vertex processing device 210 receives the original vertex list and the test stitching coefficients C(1)-C(20) from the stitching decision unit 320, modifies all texture coordinates of each vertex in the original vertex list in all lens images in a vertex-by-vertex manner, and generates a modified vertex list (to be described later); based on the modified vertex list and the four lens images from the image capture module 21, the image processing device 210 measures the regional error amounts E(1)-E(20) of all control areas R(1)-R(20) in the equidistant rectangular panoramic image and outputs the regional error amounts. Afterwards, the stitching decision unit 230 receives the regional error amounts based on the test stitching coefficients C(1)-C(20) to generate a 2D error table, and then generates the optimal stitching coefficients C(1)-C(20) based on the 2D error table. On the other hand, in a display mode, the stitching decision unit 320 is disabled, so according to the optimal stitching coefficient of the control area output by the stitching decision unit 230, the vertex processing device 210 and the image processing device 220 work together to generate the equirectangular panoramic image.

[0071] In the measurement or display mode, the vertex processing device 210 receives the target vertex P (and its data structure) from the original vertex list and twenty test / optimal joint coefficients C(1) to C(20) from the stitching decision unit 230. Then, the vertex processing device 210 interpolates two of the twenty test / optimal joint coefficients for each lens image related to the target vertex P according to the original texture coordinates of the target vertex P to generate an interpolated joint coefficient C' for each lens image. Taking the front lens image as an example, according to the original vertex list, assuming that the target vertex P has the original texture coordinates (u 1) to (c 20) in the front lens image. P ,v P ). Figure 4A An example of the relationship between the target vertex P and the ten control regions R(6)-R(15) in the front lens image is shown, wherein the ten control regions R(6)-R(15) respectively have ten test / optimal joint levels / coefficients C(6)-C(15). In one embodiment, according to the angle θ, the vertex processing device 210 processes the target vertex P (having the original texture coordinates (u P ,v P )) interpolates the two joint coefficients of the two nearest control areas to generate the interpolated joint coefficient C' of the target vertex P in the front lens image. The angle θ is clockwise and formed between a first vector V1 and a second vector V2; the first vector V1 is centered at the image center 41 (with texture coordinates (u center,v center )) as the starting point and the position 42 of the starting control area R(13) as the end point, and the second vector V2 takes the image center 41 as the starting point and the target vertex P(u P ,v P ) is the end point. Assuming θ=119.5°, since there are five control areas on the left and right sides of the front lens image, 90° / 4=22.5°, idx=θ / 22.5°=5 and θmod 22.5°=θ-idx×22.5°=7°. Therefore, the two control areas closest to vertex P are R(9) and R(10), and the interpolation joint coefficient C' of the target vertex P in the front lens image is defined / calculated by the following equation: C'=C(9)×(7 / 22.5)+C(10)×(1-7 / 22.5). The position 42 of each control area includes, but is not limited to, the gravity center of the control area, the area center of the control area, and the median point of the control area. Please note that the starting control area R(13) is only an example and is not intended to be limiting of the present invention. In another embodiment, another control area, such as R(11), may be selected as the starting control area of ​​the front lens image, as long as the angle θ is still in the clockwise direction and is from the first vector V1 (with the image center 41 (with texture coordinates (u center ,v center )) is the starting point and the position 42) pointing to the starting control area R(11) is measured to the second vector V2.

[0072] Taking the target vertex P as an example, to simplify the above calculation, the correspondence generator 23 pre-determines which two control regions (i.e., R(9) and R(10)) are the closest to the target vertex P, and writes their indexes (9 and 10) into the "joining coefficient index" field of the front lens image of the data structure of the target vertex P in the original vertex list; in addition, the correspondence generator 23 also pre-calculates the blending weight (=7 / 22.5) of the joining coefficients (C(9) and C(10)), and writes the blending weight into the "blending weight (Alpha) of the joining coefficient" field of the front lens image of the data structure of the target vertex P in the original vertex list. Please note that the test / optimal joining coefficients (C(1)-C(20)) output from the splicing decision unit 230 are arranged as a one-dimensional (1D) joining coefficient array or a one-dimensional (1D) data stream. After receiving the original vertex list, the vertex processing device 210 extracts two joining coefficients (C(9) and C(10)) from the 1D joining coefficient array output by the stitching decision unit 230 according to the "joining coefficient index" field (i.e., 9 and 10) in the data structure of the target vertex P in the front lens image, and then calculates the interpolated joining coefficient C' according to the "blending weight (Alpha) of the joining coefficient" field (i.e., 7 / 22.5) in the data structure of the target vertex P in the front lens image (please refer to Table 1).

[0073] After receiving the interpolated joint coefficient C', the vertex processing device 210 calculates the modified texture coordinates (u P ',v P '):u P '=(u P -u center )*C'+u center ;v P '=(v P -v center )*C'+v center In this way, the original texture coordinates (u2, v2) of the target vertex P in the front lens image become the modified texture coordinates (u2', v2'). In this way, according to the twenty test / optimal joint coefficients C(1) to C(20), the vertex processing device 210 sequentially modifies all the texture coordinates of the four lens images of each vertex from the original vertex list to generate the modified vertex list. Table 2 shows an example of the data structure of each vertex in the modified vertex list.

[0074] Table 2: Example of the data structure of each vertex in the modified vertex list

[0075]

[0076] After the vertex processing device 210 corrects all texture coordinates of all vertices from the original vertex list according to the test / optimal joint coefficients C(1)-C(20), the mismatch image defect problem caused by the lens center offset of the image capture module 21 (i.e., a lens center 46 has an offset distance ofs relative to the system center 43) can be greatly improved (i.e., the actual imaging position 48 will be pushed to the ideal imaging position 40), such as Figure 4B Please note that since the sphere 12 is virtual, the object 45 may be located outside, inside or on the surface of the sphere 12.

[0077] After receiving the corrected vertex list and the four lens images, the image processing device 220 generates regional error amounts E(1)-E(20) of twenty control areas R(1)-R(20) in the measurement mode, or generates an equidistant rectangular panoramic image in the display mode. Figure 5 According to an embodiment of the present invention, a schematic diagram of the image processing device is shown. Figure 5 The image processing device 220 includes a graphics engine 55, a seam engine 56, a destination buffer 510 and a temporary buffer 520. The graphics engine 55 includes a rasterization engine 560, a texture mapping circuit 570 and a measurement unit 580. The texture mapping circuit 570 includes two texture mapping engines 571-572. Figure 3B The polygon mesh is a collection of multiple quadrilaterals and / or multiple triangles. Therefore, the rasterization engine 560 can generate a quadrilateral (such as Figure 3B ) or a triangle formed by any three vertices from the modified vertex list (such as Figure 3B ) performs triangle rasterization operations on each pixel within.

[0078] In the case of quadrilateral operations, assume that the four vertices (A, B, C, D) from the modified vertex list (forming Figure 3B A quadrilateral of a polygonal mesh) is located within a range of the above twenty control areas R(1) to R(20) and is overlapped by two lens images (front and right; N=2). The four vertices (A, B, C, D) in the vertex list respectively include the following data structures: Vertex A: {(x A ,y A ),2,ID Front ,(u 1A ,v 1A ),w 1A ,ID Right ,(u2A ,v 2A ),w 2A}; Vertex B: {(x B ,y B ),2,ID Front ,(u 1B ,v 1B ),w 1B ,ID Right ,(u 2B ,v 2B ),w 2B}; Vertex C: {(x C ,y C ),2,ID Front ,(u 1C ,v 1C ),w 1C ,ID Right ,(u 2C ,v 2C ),w 2C ,}; Vertex D: {(x D ,y D ),2,ID Front ,(u 1D ,v 1D ),w 1D ,ID Right ,(u 2D ,v 2D ),w 2D The rasterization engine 560 directly performs a rasterization operation on the quadrilateral ABCD. Specifically, the rasterization engine 560 calculates the texture coordinates of each lens image for a point Q (with equidistant rectangular coordinates (x, y) and located in the quadrilateral ABCD of the polygonal mesh) by the following steps: 1. Using a bilinear interpolation method, the texture coordinates of each lens image are calculated according to the equidistant rectangular coordinates (x A ,y A ,x B ,y B ,x C ,y C ,x D ,y D ,x,y), calculate four spatial weights (a,b,c,d); 2. Calculate the sampling point Q in the front lens image F Texture coordinates (corresponding to point Q): (u1,v1) = (a*u 1A +b*u 1B +c*u 1C +d*u 1D ,a*v 1A +b*v 1B +c*v1C +d*v 1D );Calculate the sampling point Q in the right lens image R Texture coordinates (corresponding to point Q): (u2, v2) = (a*u 2A +b*u 2B +c*u 2C +d*u 2D ,a*v 2A +b*v 2B +c*v 2C +d*v 2D ). Finally, the rasterization engine 560 transmits the two texture coordinates (u1, v1) and (u2, v2) in parallel to the two texture mapping engines 571-572. Wherein, a+b+c+d=1. According to the two texture coordinates (u1, v1) and (u2, v2), the two texture mapping engines 571-572 use any suitable method (such as nearest-neighbor interpolation, bilinear interpolation, or trilinear interpolation) to texture map the texture data of the front and right lens images to generate two sample values ​​s1 and s2. Wherein, each of the sample values ​​can be a brightness (luma) value, a chroma (chroma) value, an edge (edge) value, a pixel color value (RGB) or a motion vector (motion vector).

[0079] In the case of operating on a triangle, the rasterization engine 560 and the two texture mapping engines 571-572 form a triangle (such as Figure 3B ) performs a similar operation (similar to the case of the quadrilateral) to generate two corresponding sample values ​​s1 and s2, except that the rasterization engine 560 uses a barycentric weighting method in step (1) instead of the above-mentioned bilinear interpolation method to calculate the equidistant rectangular coordinates (x A ,y A ,x B ,y B ,x C ,y C ,x,y), calculate the three spatial weights (a,b,c).

[0080] In this measurement mode, the measurement unit 580 measures / estimates the regional error amounts E(1)-E(20) of the twenty control areas R(1)-R(20) in the equidistant rectangular panoramic image according to the equidistant rectangular coordinates (x, y) of the point Q and the two sampling values ​​(s1-s2). According to the equidistant rectangular coordinates (x, y) of the point Q, the measurement unit 580 determines whether the point Q falls into the twenty control areas (see Figure 3C ), if it is determined that the point Q falls into any control area, the regional error amount of the control area is estimated / measured. The measuring unit 580 can use any known algorithm, such as the sum of absolute differences (SAD), the sum of squared differences (SSD), the median absolute deviation (MAD), etc., to estimate / measure the regional error amount of the control area. For example, if it is determined that the point Q falls into the control area R(11), the measuring unit 580 uses the following equations: E=|s1-s2|; E(11)+=E to accumulate the absolute value of the difference in sample values ​​between each point of the control area R(11) in the right lens image and the corresponding point of the control area R(11) in the front lens image, so as to obtain a SAD value as the regional error amount E(11) of the control area R(11). In this way, the measuring unit 580 measures the regional error amounts E(1) to E(20) of the twenty control areas R(1) to R(20).

[0081] One of the features of the present invention is that in a predetermined number of loops ( Fig. 6A The predetermined number of loops is related to an offset ofs, which is the distance from the lens center 46 of the image capture module 21 to the system center 43 (refer to Figure 4B ).

[0082] Please go back Figure 2 , in measurement mode, according to Figure 4B The stitching decision unit 230 sets the twenty joint coefficients C(1) to C(20) to different value ranges to measure the regional error amount, and sets the twenty joint coefficients to the same value each time (or each loop). For example, assuming ofs = 3 cm, the twenty joint coefficients C(1) to C(20) are set to the value range of 0.96 to 1.04. If the increment is 0.01 each time, there will be a total of nine measurements; assuming ofs = 1 cm, the twenty joint coefficients C(1) to C(20) are set to the value range of 0.99 to 1.00. If the increment is 0.001 each time, there will be a total of ten measurements.

[0083] Fig. 6A According to one embodiment of the present invention, a flow chart of a method for determining the optimal bonding coefficient of all control areas is shown. Figure 4B , 5 6A, assuming ofs=3 cm, illustrates the method of determining the optimal splicing coefficients of all control areas of the present invention (applicable to the splicing decision unit 230 in the measurement mode).

[0084] Step S602: Set the number of loops Q1 and the test joint coefficient to new values. In one embodiment, Q1 is set to 1 in the first loop, and Q1 is increased by 1 in each subsequent loop; if ofs = 3 cm, the test joint coefficients C(1) to C(20) are all set to 0.96 (i.e., C(1) = ... = C(20) = 0.96) in the first loop, and in subsequent loops, the joint coefficients C(1) to C(20) are set to 0.97, ..., 1.04 in sequence.

[0085] Step S604: Clear all regional error amounts E(m) to 0, where m=1, 2, ..., 20.

[0086] Step S606: After the vertex processing device 210 completes the operation according to the test joint coefficient set in step S602, the image processing device 220's image engine 56 measures / obtains the regional error amounts E(1) to E(20) of the twenty control areas R(1) to R(20) of the equidistant rectangular panoramic image. For ease of description, this step S606 is represented by E(m)=f(C(m)), where m=1, 2, ..., 20, and f() represents (by the image engine 56) measuring the regional error amount E(m) according to the corresponding test joint coefficient C(m) and the four lens images output by the image capture module 21.

[0087] Step S608: Store all regional error values ​​E(1)-E(20) in a two-dimensional (2D) error table. Table 3 shows an example of the 2D error table when ofs=3 cm (the numerical range of the test joint coefficient is 0.96-1.04).

[0088] Table 3: Example of 2D error table when ofs = 3 cm (value range of test joint coefficient 0.96~1.04)

[0089]

[0090] In the 2D error table of Table 3, there are a total of twenty regional error amounts and nine different test joint coefficients. Due to space limitations, Table 4 only shows eight regional error amounts and six different test joint coefficients, and the rest are temporarily skipped. Please note that the number of regional error amounts and test joint coefficients in the above 2D error table is only an example, not a limitation of the present invention. In actual implementation, the above 2D error table can use any number of regional error amounts and joint coefficients.

[0091] Step S610: Determine whether the number of loops Q1 reaches the upper limit of 9. If yes, jump to step S612, otherwise, return to step S602.

[0092] Step S612: Perform coefficient decision operation according to the above 2D error table.

[0093] Step S614: Output the optimal joint coefficient C(m), wherein m=1, 2, ..., 20. In the display mode, the optimal joint coefficient C(m) is output to the vertex processing device 210, so that the image processing device 220 generates a corresponding equidistant rectangular panoramic image.

[0094] Figure 6B According to an embodiment of the present invention, a flow chart of a method for the splicing decision unit 230 to perform the coefficient decision operation of step S612 is shown. Figure 3C and 6A -6B, describes all the steps for performing the coefficient decision operation.

[0095] Step S661: Set Q2 to 0 for initialization.

[0096] Step S662: Extract a selected decision group from the 2D error table. Figure 3C and 7B , usually each control area is adjacent to two control areas, and a selected control area and its two adjacent control areas form a selected decision group to determine the optimal joining coefficient of the selected control area. For example, a selected control area R(9) and its two adjacent control areas R(8) and R(10) form a decision group. However, if a selected control area (such as R(1)) is located at the top or bottom of the overlapping area, the selected control area R(1) will only form a decision group with its only adjacent control area R(2) to determine its optimal joining coefficient C(1). The description of the subsequent steps assumes that a control area R(2) is selected, and R(2) and its two adjacent control areas R(1) and R(3) form a selected decision group to determine its optimal joining coefficient C(2).

[0097] Step S664: Determine a local minimum value in the regional error values ​​of each control region of the selected decision group. Table 4 shows an example of the regional error values ​​and test joint coefficients of R(1) to R(3).

[0098] Table 4: Examples of regional errors and test joint coefficients for R(1) to R(3)

[0099]

[0100]

[0101] As shown in Table 4, there is only one local minimum value among the nine regional error values ​​of R(1), while there are two local minimum values ​​among the nine regional error values ​​of R(2) and R(3), respectively. An asterisk (*) is marked next to each local minimum value in Table 4.

[0102] Step S666: Select candidates based on the local minimum values. Table 5 shows the candidates selected from the local minimum values ​​in Table 4, where ID represents index, WC represents joint coefficient, and RE represents regional error. The number of candidates is equal to the number of local minimum values ​​in Table 4.

[0103] Table 5: Examples of selecting candidates from the local minimum values ​​in Table 4

[0104]

[0105] Step S668: Create a link metric based on the candidates in Table 5. Figure 6C As shown, a link metric is established based on the candidates in Table 5.

[0106] Step S670: Determine the minimum sum of the link metrics among all paths of the link metric. and The minimum value between the two About two linked measurements and The minimum value between the two Then, the sum of the link metrics of path 0-0-0 and path 0-1-1 is calculated as follows: and because So it can be decided is the minimum sum of the link measurement values ​​in all paths of the link measurement, such as Figure 6C The solid line path in .

[0107] Step S672: Determine the optimal bonding coefficient for the selected control area. is the minimum sum of the link metrics of all paths, so 1.02 is determined as the optimal joint coefficient of the control area R(2). However, if the sum of the link metrics of two or more paths is the same at the end of the calculation, the joint coefficient of the node with the minimum regional error is selected as the optimal joint coefficient of the selected control area. Here, the value of the number of loops Q2 is incremented by 1.

[0108] Step S674: Determine whether the number of loops Q2 reaches the upper limit of 20. If so, end this process, otherwise return to step S662 to process the next control area.

[0109] The definitions of the relevant terms mentioned throughout the specification and subsequent claims are as follows, unless otherwise specified in this specification. The term "face image" refers to an image generated by the projection of a corresponding lens image from the image capture module 21; the projection is, for example, an equirectangular projection, a cylindrical projection, a Miller projection, a Mercator projection, a Lambert cylindrical equal area projection, or a Pannini projection. The term "seam area" refers to a projection image that is located between two adjacent first main images and second main images and is overlapped by a seam image A and a seam image B, wherein the first main image and the seam image A are part of a first working image, and the second main image and the seam image B are part of a second working image. Generally speaking, the face image is divided into three parts for subsequent stitching processing, for example, Fig. 7A In the example, the first part of the left working surface image is located in the seam area S(2) or on the right side of the left main image, which is called seam image A; the second part of the left working surface image is located in the seam area S(1) or on the left side of the left main image, which is called seam image B; the third part of the left working surface image is located between the seam areas S(1) and S(2), which is called the left main image. The same segmentation method is also applicable to the right working surface image, the front working surface image, and the back working surface image.

[0110] Fig. 7A An example showing an equirectangular panoramic image consisting of four main images and four seam areas. Figure 7B show Fig. 7A An enlarged view of the seam area S(2). Figure 7B The quadrilateral 72 in Figure 3B The quadrilaterals of the polygonal mesh and the intersection points 71 of the quadrilaterals 72 are vertices. According to the FOV of the lens in the image capture module 21, the resolution of the lens sensor and the angle of the lens, the overlap area (e.g. Figure 1BThe size of A(1)~A(4)) in the figure will change accordingly. According to the present invention, P1 control areas are located in a seam area, and the seam area is located in a corresponding overlapping area, where P1>=3. Generally speaking, the width of the overlapping area is greater than or equal to the width of the seam area, and the width of the seam area is greater than or equal to the width of the P1 control areas. For example, the seam area S(2) is located in the overlapping area A(2) (not shown), the width of A(2) is greater than or equal to the width of S(2), and the width of S(2) is greater than or equal to the width of R(6)~R(10). In a preferred embodiment, the ratio of the widths of the overlapping area (such as A(2)), the seam area (such as S(2)) and the control area (such as R(6)~R(10)) is 7:5:1. In another embodiment, the ratio of the widths of the overlapping area (such as A(2)), the seam area (such as S(2)) and the control area (such as R(6) to R(10)) is 9:7:1, 9:9:1, 9:5:1, 7:7:1 or 5:5:1.

[0111] To produce Fig. 7A The image engine 56 generates four working surface images according to the equirectangular projection of the four lens images from the image capture module 21. In actual operation, nine images related to the above four working surface images are generated, namely, a primary image (including a back main image, a front main image, a left main image and a right main image) and eight seam images (including four seam images A and four seam images B). In order to store the above nine images in nine different buffers for subsequent stitching operations, in the display mode, the correspondence generator 23 generates (m1+8) original single lens vertex lists, namely, m1 original primary vertex lists and eight original seam vertex lists corresponding to the above four working surface images and the eight seam images, respectively, where 1<=m1<=4. For example, if m1=1, one original main vertex list is used to generate four main images at a time; if m1=2, any one of the two original main vertex lists is used to generate two main images at a time; if m1=4, any one of the four original main vertex lists is used to generate one of the four main images at a time. Correspondingly, in order to minimize the problem of mismatched image defects caused by the lens center offset of the image capture module 21, the vertex processing device 210 generates (m1+8) modified single-lens vertex lists, i.e., m1 modified main vertex lists and eight modified seam vertex lists, corresponding to the above-mentioned four working surface images and eight seam images, respectively, based on the (m1+8) original single-lens vertex lists and the above-mentioned twenty optimal joint coefficients C(1)~C(20). Please note that the format of each of the (m1+8) original single-lens vertex lists is similar to Table 1, but only contains a set of parameters for a single lens, i.e., ID k 、(u k ,v k )、(idxk0 ,idx k1 ) and Alpha k , where k = 0, 1, 2 or 3. Note that the format of each of the (m1+8) modified single-lens vertex lists is similar to Table 2, but only contains a set of parameters for a single lens, namely ID k and (u' k ,v' k ), where k = 0, 1, 2 or 3.

[0112] In the display mode, the image engine 56 performs rasterization and texture mapping based on the m1 modified primary vertex lists and the four lens images to generate the primary images (including the back primary image, the front primary image, the left primary image, and the right primary image). Then, based on the eight modified seam vertex lists and the four lens images, the image engine 56 performs rasterization and texture mapping "sequentially" to generate the four seam images A and the four seam images B, that is, one seam image is generated at a time. For example, the image engine 56 performs rasterization and texture mapping based on a corresponding modified seam vertex list and a front lens image to generate one seam image A related to the front working surface image at a time. In another embodiment, the image engine 56 may generate the images in the opposite order, that is, the eight seam images may be generated first, and then the primary image may be generated.

[0113] There is a special case that when the image capture module 21 includes a wide-angle camera, the leftmost working surface image and the rightmost working surface image of the equirectangular projection image both include only a main image and a seam image, that is, one of the seam image A and the seam image B of the leftmost working surface image and the rightmost working surface image will be merged into the main image. For example, assuming that the image capture module 21 includes a dual-lens wide-angle camera (such as 180°×60°) to generate a left lens image and a right lens image, the image engine 56 generates a primary image (including a left main image and a left main image) and a seam image A and a seam image B related to a single seam area according to the two lens images, one or two modified main vertex lists and two modified seam vertex lists. Finally, the image processing device 220 combines the primary image, the seam image A and the seam image B into a wide-angle image.

[0114] Figure 7C According to an embodiment of the present invention, a flow chart of a method for determining a minimum error path (or a seam) in a seam region is shown. Figure 5 and 7A -7D, illustrates the method for determining a seam region's minimum error path (performed by the seam engine 55 of the image processing device 220 in the display mode).

[0115] For ease of explanation, the coordinates (x, y) refer to the position of a pixel in an equirectangular projection.

[0116] Step S702: In the same seam region, calculate the absolute value of the pixel value difference between each pixel position in the seam image A and the adjacent pixel position in the seam image B. In one embodiment, the following program code is provided to calculate the absolute value of the pixel value difference |A(x,y)-B(x+1,y)| between each pixel position (x,y) in the seam image A and the adjacent pixel position (x+1,y) in the seam image B in the same seam region (ranging from Y0 to Y1 and X0 to X1), where A(x,y) represents the pixel value at position (x,y) in the seam image A related to a first working surface image, and B(x+1,y) represents the pixel value at position (x+1,y) in the seam image B related to a second working surface image. Please note that the two pixels A(x,y) and B(x+1,y) must be located on the same row y, and the first working surface image and the second working surface image overlap in the seam region.

[0117] for(y=Y0;y<=Y1;y++){

[0118] for(x=X0;x<=X1;x++){

[0119] Err[y][x]=|A(x,y)–B(x+1,y)|; / / Array Err[y][x] stores each pixel A(x,y) in the seam image A

[0120] The absolute value of the pixel value difference relative to the adjacent right pixel B(x+1,y) in the seam image B

[0121] }

[0122] }

[0123] In another embodiment, the squared error of the pixel value of each pixel position in the seam image A relative to the adjacent pixel position in the seam image B is calculated instead of the absolute value of the pixel value difference or error.

[0124] Step S704: In the seam area, the minimum cumulative error (or the absolute value of the minimum cumulative pixel value difference) of each pixel in a current row is calculated, and the X coordinate with the minimum cumulative error in the previous row is found. In one embodiment, the following program code is provided to calculate the minimum cumulative error of each pixel in a current row y, and find the X coordinate with the minimum cumulative error in the previous row (y-1), where y ranges from Y0 to Y1.

[0125]

[0126] Step S706: Find the X coordinate with the minimum cumulative error in the bottom row of the seam area. In one embodiment, the following program code is provided to find the X coordinate with the minimum cumulative error in the bottom row of the seam area.

[0127]

[0128]

[0129] Step S708: Trace / backtrack from the bottom row of the seam area to the top row to form a path (or a seam) with minimum error. Fig.7D As shown, a seam 78 spans the seam area S(2) and has a pixel position, i.e., an x-coordinate, in each row. The seam 78 is a collection of a series of separation points (or seam points) on all rows of the seam area. In one embodiment, the following program code is provided to form a seam in a seam area:

[0130]

[0131] At the end of this step, the array Seam[y] will contain a series of separation points on all rows (Y0-Y1) of the seam area, i.e., the minimum error path or the seam. In another embodiment, a person skilled in the art can appropriately modify the contents of steps S704-S708 to form the seam of the seam area in the reverse order (tracing from the top row of the seam area to the bottom row), which also falls within the scope of the present invention.

[0132] Fig. 8A According to an embodiment of the present invention, a flowchart of an image processing method is shown. Figure 2 , 5 8A-8D illustrate the image processing method (applicable to the projection image processing system 200 of the present invention).

[0133] Step S802: Obtain a 2D error table with all test joint coefficients and regional error values ​​of all control areas. Fig. 6A At the end of step S802, a 2D error table (such as Table 3) including all test joint coefficients and regional error amounts E(1) to E(20) of all control areas R(1) to R(20) can be obtained. Please note that all test joint coefficients are related to the offset ofs of the lens center 46 of the image capture module 21 from its system center 43, such as Figure 4B shown.

[0134] Step S804: Determine the optimal bonding coefficients for all control areas based on the 2D error table. Fig. 6A Steps S612 to S614 and Figure 6B At the end of step S804, the optimal joint coefficients C(1) to C(20) of all control areas R(1) to R(20) are obtained and transmitted to the vertex processing device 210; at the same time, the measurement mode is ended and the imaging mode is started.

[0135] Step S806: Generate each work surface image according to the multiple lens images, the multiple original single lens vertex lists and the best joint coefficient obtained in step S804. As described above, the vertex processing device 210 generates multiple modified single lens vertex lists (including at least one modified main vertex list and multiple modified seam vertex lists) in sequence according to the multiple original single lens vertex lists (including at least one original main vertex list and multiple original seam vertex lists) output by the correspondence generator 23 and the best joint coefficient obtained in step S804. Fig. 7A In the example of an equidistant rectangular panoramic image with four seam areas, in measurement mode, Figure 5 The image engine 56 generates four working surface images (including four main images, four seam images A and four seam images B) according to the four lens images from the image capture module 21, the (m1+8) corrected single lens vertex lists and the twenty best joint coefficients C(1)-C(20) obtained in step S804, where 1<=m1<=4. After generating the four working surface images, the image engine 56 stores nine images (including one primary image (including the four main images), the four seam images A and the four seam images B) in one of the following two ways. Method 1: First, the image engine 56 stores the nine images in nine different temporary buffers 520, and the nine images do not overlap each other; then, the primary image is transmitted to a destination buffer 510, wherein the four seam areas are empty; finally, the seam engine 55 generates four spliced ​​seam images according to the four seam images A and the four seam images B, and then stores the four spliced ​​seam images in the four seam areas of the destination buffer 510 for display. Method 2: First, the image engine 56 directly stores the primary image in the destination buffer 510 and stores the four seam images A and the four seam images B in eight different temporary buffers 520 respectively; then, the seam engine 55 generates four spliced ​​seam images according to the four seam images A and the four seam images B, and then stores the four spliced ​​seam images in the four seam areas of the destination buffer 510. For example, a seam image A which is a part of the left working surface image and a seam image B which is a part of the front working surface image are fed into the seam engine 55 for splicing, and after a spliced ​​seam image is generated, the spliced ​​seam image is stored in the seam area S(2) of the destination buffer 510, as shown in FIG. Fig. 7A shown.

[0136] Assuming that the image capture module 21 includes a wide-angle camera with three lenses and generates three lens images (ie, a left lens image, a front lens image, and a right lens image), Figure 5 The image engine 56 generates three working surface images (including three main images and two seam images A and two seam images B related to two seam areas) according to the three lens images, at least five modified single lens vertex lists and a plurality of optimal joint coefficients. After generating the three working surface images, the image engine 56 stores five images (including one primary image (including the three main images), the two seam images A and the two seam images B) using the first or second method for subsequent image stitching.

[0137] Step S808: Determine the path (or seam) that minimizes the error in each seam area. Fig. 7A The four seam areas S(1) to S(4) are executed Figure 7C At the end of step S808, four minimum error paths or seams of the four seam regions S(1)-S(4) are generated.

[0138] Step S810: splice the overlapping seam images A and B together according to the minimum error path or seam of each seam area. Fig.7D In the example of FIG. 5 , the seam 78 is used to stitch the seam image A related to the left working surface image and the seam image B related to the front working surface image together to generate a stitched seam image. Finally, a total of four stitched seam images are respectively stored in the four seam areas S(1) to S(4) of the destination memory 510. Therefore, the four stitched seam images and the above four main images form a complete equidistant rectangular panoramic image.

[0139] Figure 8B An example of a section of a seam area is shown, and the seam area is associated with a seam 78, a seam image A, and a seam image B. Figure 8C An example of displaying the size of a seam area and the size and different positions of three rows of update windows 81. In step S810, the present invention provides the following three methods (1) to (3) to stitch seam images A and B to generate a corresponding stitched seam image. (1) Traditional method: stitch the corresponding seam image A and seam image B together according to the minimum error path or seam of each seam area. Figure 8BAs shown, for each of the stitched seam images or seam areas, if a pixel position (x, y) is located on the left side of the seam 78, its pixel value in the seam image A is designated as the pixel value of the same pixel position (x, y) in the stitched seam image; conversely, if a pixel position (x, y) is located on the right side of the seam 78, its pixel value in the seam image B is designated as the pixel value of the same pixel position (x, y) in the stitched seam image.

[0140] (2) Seam blending method (each seam area is a continuous whole area and is not cut): According to the seam 78 of a seam area, an update window 81 is applied to each row of the seam area to splice the corresponding seam image A and seam image B. In order to smooth the pixel value transition near the seam 78, the present invention provides a seam blending method to update the pixel value in an update window 81, wherein the width of the update window 81 is equal to 2×Ra and a separation point 82 on the seam 78 is located at the center of the update window 81, such as Figure 8C As shown. For each row of the seam area S(2), if a pixel position (x, y) is located on the left side of the update window 81, its pixel value in the seam image A is designated as the pixel value of the same pixel position (x, y) in the stitched seam image; on the contrary, if a pixel position (x, y) is located on the right side of the update window 81, its pixel value in the seam image B is designated as the pixel value of the same pixel position (x, y) in the stitched seam image; further, if a pixel position (x, y) is located within the update window 81, a blending weight is calculated according to the final width of the update window 81 and the distance of the pixel position (x, y) relative to the left edge of the update window 81, and then the blended pixel value of the same pixel position (x, y) in the stitched seam image is calculated according to the blending weight and the two corresponding pixel values ​​of the seam image A and the seam image B (i.e., ImgA(x, y) and ImgB(x, y)). In one embodiment, the following program code is provided to update the pixel value in the update window 81 according to the final width of the update window 81 in each row of the seam area and the position of each pixel position (x, y) relative to the update window 81.

[0141]

[0142] (3) Wrap around seam mixing method (one of the above seam areas is divided into two discontinuous parts and located on opposite sides of the panoramic image): In a special case, a seam area may be located near the left edge and the right edge of the equidistant rectangular panoramic image at the same time. Fig.8DAs shown, the seam area S(3) is located at the leftmost and rightmost sides of the equidistant rectangular panoramic image. In other words, the seam area S(3) is divided into two parts. To handle this situation, the present invention provides a surrounding seam blending method. In one embodiment, the following program code is provided to form a stitched seam image of a seam area (ranging from Y0 to Y1 and X0 to X1) according to the surrounding seam blending method:

[0143]

[0144]

[0145] Therefore, even if Fig.8D The seam area S(3) is located at the leftmost and rightmost sides of the equidistant rectangular panoramic image. The above-mentioned surrounding seam blending method can still generate a correct stitched seam image.

[0146] The compensation device 22 and the correspondence generator 23 of the present invention can be implemented by software, hardware, or a combination of software (or firmware) and hardware. An example of a simple solution is a field programmable gate array or an application specific integrated circuit. In a preferred embodiment, the vertex processing device 210 and the image processing device 220 are implemented using a graphics processing unit (GPU) and a first program memory; the splicing decision unit 230 and the correspondence generator 23 are implemented using a first general-purpose processor and a second program memory. The first program memory stores a first processor executable program, and the second program memory stores a second processor executable program. When the graphics processing unit executes the first processor executable program, the graphics processing unit is configured to operate as: the vertex processing device 210 and the image processing device 220. When the first general-purpose processor executes the second processor-executable program, the first general-purpose processor is configured to operate as: the splicing decision unit 230 and the correspondence generator 23 .

[0147] In another embodiment, the compensation device 22 and the correspondence generator 23 are implemented by a second general-purpose processor and a third program memory. The third program memory stores a third processor executable program. When the second general-purpose processor executes the third processor executable program, the second general-purpose processor is configured to operate as: the vertex processing device 210, the splicing decision unit 230, the correspondence generator 23 and the image processing device 220.

[0148] The above are only preferred embodiments of the present invention and are not intended to limit the scope of the patent application of the present invention; any other equivalent changes or modifications that do not deviate from the spirit disclosed by the present invention should be included in the patent scope of this application.

Claims

1. An image processing method, characterized in that: The method is used to receive M lens images to generate a projection image, and the method includes: Determining P optimal stitching coefficients of P control areas in the projection image according to a 2D error table and M lens images, wherein an image capture module is used to capture a field of view having a horizontal field of view of X degrees and a vertical field of view of Y degrees to generate the M lens images; Generate M working surface images according to a first vertex list, the P optimal joint coefficients and the M lens images; Determine N seams of N seam areas; and A stitching step: stitching two overlapping seam images of each seam area according to the seams of each seam area to generate a stitched seam image; Wherein, each working surface image includes z joint images and one main image, and each joint area is located between two adjacent main images in the projection image; The 2D error table includes a plurality of test joint coefficients and a plurality of corresponding accumulated pixel value errors in the P control areas; The P control areas are respectively located within the N seam areas, and the N seam areas are respectively located within the N overlapping areas of the M working surface images; The first vertex list includes a plurality of first vertices having a first data structure, and the first data structure defines a first vertex mapping between a single lens image and the projection image; and Among them, 1<=z<=2, M>=2, N>=1, P>=3, X<=360 and Y<180.

2. The method according to claim 1, characterized in that Also includes: The projection image is formed according to the N spliced ​​seam images of the N seam areas and the M main images.

3. The method according to claim 1, characterized in that The step of determining the P optimal bonding coefficients of the P control zones comprises: determining the test joint coefficient according to an offset of a lens center of the image capture module relative to a system center of the image capture module; A correction step: correcting the texture coordinates of each lens image in the second data structure of each second vertex in a second vertex list according to the test joint coefficient to generate a third vertex list; Measuring step: measuring the accumulated pixel value errors in the P control areas according to the third vertex list and the M lens images; Repeating the correction step and the measurement step until all the test joint coefficients are processed to form the 2D error table; and Determine the optimal joint coefficient of each control area according to at least a local minimum value of the accumulated pixel value error of one or two nearest control areas of each control area in the 2D error table; The second vertex list includes a plurality of second vertices having a second data structure, and the second data structure defines a second vertex mapping between the M lens images and the projection image; and The third vertex list includes a plurality of third vertices having a third data structure, and the third data structure defines a third vertex mapping between the M lens images and the projection image.

4. The method according to claim 1, characterized in that: The M working surface images are derived from a preset projection of a corresponding lens image of the image capture module.

5. The method according to claim 4, characterized in that The preset projection is one of the equirectangular projection, the Miller projection, the Mercator projection, the Lambert cylindrical equal area projection and the Panini projection.

6. The method according to claim 1, characterized in that If z=2, the main image of each working surface image is located between the z seam images.

7. The method according to claim 1, characterized in that The first vertex list includes m1 first main vertex lists and 2×N first seam vertex lists, and the step of generating the M working surface images includes: According to the P optimal joint coefficients, sequentially modify the texture coordinates of all first vertices in the first vertex list in a single lens image to generate a fourth vertex list, wherein the fourth vertex list includes m1 second main vertex lists and 2×N second seam vertex lists, where 1<=m1<=M; and According to the M lens images, performing rasterization and texture mapping operations on each point in each polygon formed by a set of fourth vertices from the fourth vertex list, so as to sequentially generate M main images, N first seam images, and N second seam images; wherein the m1 first main vertex lists and the 2×N first seam vertex lists include the first vertices having a first data structure, and the first data structure defines a first vertex mapping between the single lens image and the projection image; and The m1 second main vertex lists and the 2×N second seam vertex lists include a plurality of fourth vertices having a fourth data structure, and the fourth data structure defines a fourth vertex mapping between a corresponding lens image and the projection image.

8. The method according to claim 7, characterized in that The step of correcting the texture coordinates comprises: Modify the texture coordinates of the single lens image in the first data structure of each first vertex according to the test joint coefficient and two joint coefficient indices and a blending weight of the single lens image in the first data structure of each first vertex to generate the fourth vertex list; The two joining coefficient indexes are used to select two of the test joining coefficients, and the blending weight is used to blend the two selected test joining coefficients.

9. The method according to claim 8, characterized in that The step of correcting the texture coordinates further comprises: Calculate an interpolated joint coefficient of the target first vertex in the single lens image according to the test joint coefficient and the two joint coefficient indices of the single lens image in the first data structure of a target first vertex and the blending weight; Calculating a modified texture coordinate of the target first vertex in the single lens image according to the interpolation joint coefficient, the interpolation joint coefficient of the target first vertex in the single lens image, and the texture coordinate; and The modified texture coordinates of the target first vertex in the single lens image are stored in a fourth data structure corresponding to a fourth vertex in the fourth vertex list.

10. The method according to claim 7, characterized in that The step of generating the M working surface images further comprises: storing the M main images in a first buffer for display; and The N first seam images and the N second seam images are stored in 2×N different second buffers, wherein the 2×N second buffers are separated from the first buffer.

11. The method according to claim 7, characterized in that The step of generating the M working surface images further comprises: storing the M main images, the N first seam images, and the N second seam images in (m2+2×N) different first buffers, where m2<=M; and The M main images are moved to a second buffer for display.

12. The method according to claim 1, characterized in that The step of determining N seams in N seam areas comprises: In a target seam area, calculating a pixel value error of each first pixel position in a third seam image relative to a corresponding second pixel position in a fourth seam image, and storing the error in a fifth data structure, wherein each first pixel position and its corresponding second pixel position are located in the same row and adjacent to each other; According to the data of the fifth data structure and a preset search range, the minimum cumulative pixel value error of each first pixel position in the current row y of the target seam area is calculated to store in a sixth data structure, and the x coordinate with the minimum cumulative pixel value error in the previous row (y-1) is found to store in a seventh data structure; According to the sixth data structure, finding the x-coordinate with the minimum cumulative pixel value difference in the bottom row of the target seam area; and According to the seventh data structure, the x coordinates of the bottom row of the target seam area are traced back to the top row to form the seam of the target seam area, and the x coordinates of each row penetrated are stored in an eighth data structure.

13. The method according to claim 1, characterized in that The two overlapping seam images include a fifth seam image and a sixth seam image, wherein the fifth seam image is related to a first working surface image of the M working surface images and the sixth seam image is related to a second working surface image of the M working surface images, wherein the main image of the first working surface image and the main image of the second working surface image are respectively located on the left and right sides of a corresponding seam area, and the seam of the corresponding seam area includes multiple separation points, which are respectively located in each row of the corresponding seam area.

14. The method according to claim 13, characterized in that The splicing step includes a specified step: For each row of the corresponding seam area, a pixel value is assigned to the pixel position in the spliced ​​seam image according to the relative position of a pixel position in the spliced ​​seam image and the corresponding separation point located on the same row, the fifth seam image and the sixth seam image.

15. The method according to claim 14, characterized in that The specifying step comprises: If the pixel position is located on the left side of the seam, assigning the pixel value of the pixel position in the fifth seam image to the same pixel position in the stitched seam image; If the pixel position is located on the right side of the seam, assigning the pixel value of the pixel position in the sixth seam image to the same pixel position in the stitched seam image; and If the pixel position is located on the seam, a corresponding pixel value of the pixel position in one of the fifth seam image and the sixth seam image is assigned to the same pixel position in the stitched seam image.

16. The method according to claim 13, characterized in that The splicing step comprises: splicing the two overlapping seam images by providing an update window to each row of the corresponding seam area, according to the seam of the corresponding seam area and whether one of the N seam areas is divided into two separate parts; Wherein, the separation point on each row of the corresponding seam area is located at the center of the update window.

17. The method according to claim 16, characterized in that The splicing step includes a specified step: For each row of the corresponding seam area, If a pixel position is located between the left edge of the seam area and the left edge of the update window, assigning the pixel value of the pixel position in the fifth seam image to the same pixel position in the stitched seam image; If the pixel position is located between the right edge of the seam and the right edge of the update window, assigning the pixel value of the pixel position in the sixth seam image to the same pixel position in the stitched seam image; as well as If the pixel position is located in the update window, a blending weight is calculated according to a final width of the update window and a distance of the pixel position relative to a left edge of the update window, and a blended pixel value of the same pixel position in the spliced ​​seam image is calculated according to the blending weight and corresponding pixel values ​​of the pixel position in the fifth seam image and the sixth seam image.

18. The method according to claim 17, characterized in that Each seam area is a complete area and the stitching step further comprises: Before the specifying step and the calculating of the blending weight and the blending pixel value, for each row of the corresponding seam area, Determining the coordinates of the left edge and the right edge of the update window according to the original width of the update window; and If one of the left edge and the right edge of the update window extends out of the corresponding seam area, the edge of the update window is corrected to align with the boundary of the corresponding seam area to obtain the final width of the update window.

19. The method according to claim 17, characterized in that One of the N seam areas is divided into two separate parts and the two separate parts are located at the leftmost and rightmost sides of the projection image, wherein the stitching step comprises: Before the specifying step and the calculating of the blending weight and the blending pixel value, for each row of the corresponding seam area: Subtract the x-coordinate of the left edge of the corresponding seam area from the x-coordinate of a corresponding separation point to obtain a relative distance of the corresponding separation point; If the relative distance is less than 0, adding the width of the projection image to the relative distance to correct the relative distance; and Determine the positions of the left edge and the right edge of the update window according to the relative distance and the original width of the update window; If one of the left edge and the right edge of the update window extends beyond the corresponding seam area, correcting the edge of the update window to align with the boundary of the corresponding seam area to obtain a final width of the update window; calculating a distance between a pixel position in the corresponding seam area and a left edge of the projection image; and If the distance of the pixel position in the corresponding seam area relative to the left edge of the projection image is greater than the width of the projection image, the distance of the pixel position relative to the left edge of the projection image is subtracted from the width of the projection image to correct the pixel position.

20. The method according to claim 1, characterized in that Each seam area includes P1 control areas arranged in a row, and P1>=3.

21. The method according to claim 20, characterized in that For a seam area located in an overlap area and the P1 control areas located in the seam area, the width of the overlap area is greater than or equal to the width of the seam area, and the width of the seam area is greater than or equal to the width of the P1 control areas.

22. The method according to claim 1, characterized in that The projected image is one of a panoramic image and a wide-angle image.

Images