Image processing system and method

Abstract

The present application provides an image processing system and method. The system includes a multi-lens camera, a vertex list generator and an image processing device. The multi-lens camera is used to capture a field of view covering X degree horizontal view and Y degree vertical view, to generate a plurality of lens images, wherein X<=360 and Y<=180. The vertex list generator is used to generate a first main vertex list according to a corresponding table, and generate a first ROI vertex list according to the first main vertex list and position information of a region of interest (ROI) when the ROI overlaps with at least one measurement region in an overlapping area of a projection image. The image processing device is used to generate the projection image according to the plurality of lens images and the second main vertex list related to the first main vertex list in a display mode.

Description

Technical Field

[0001] This invention relates to image processing, and more particularly to an image processing system and method for generating regions of interest (ROIs). Background Technology

[0002] A multi-lens camera captures a field of view that encompasses a 360-degree horizontal field of view (FOV) and up to 180 degrees of vertical field of view. For example, such as... Figure 1A As shown, a known camera comprises four lenses mounted on four working faces (right, left, front, and back) of a cubic structure 11. Any two adjacent faces of these four working faces are substantially orthogonal; for example, the four working faces are oriented towards the longitudes of a virtual sphere 12 at 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively, to simultaneously capture a field of view with a 360-degree horizontal FOV and a 90-degree vertical FOV, thereby generating images from four lenses. Please refer to... Figure 1B The pixels in regions A(1) to A(4) are formed by the overlap of two adjacent lens / texture images, while the pixels in region 15 come from a single lens / texture image. Therefore, the multiple overlapping regions A(1) to A(4) must be stitched together to form a panoramic image.

[0003] U.S. Patent No. 10,810,700B2 (the contents of which are incorporated herein by reference in their entirety as part of this specification) discloses a known panoramic image stitching method for minimizing image mismatch defects caused by lens center offset in the image capturing module, for example: Figure 4C In the image capturing module 21, the center 46 of one lens has an offset of 0s relative to the system center 43. However, when a face appears in an overlapping area A(2) and is accompanied by a potted plant (such as... Figure 1B As shown, the panoramic image generated by the aforementioned known panoramic image stitching method exhibits a significant defect: when the face is a distant object and the potted plant is a near object, the overlapping area A(2) of the panoramic image is likely to show a potted plant with a missing corner and two overlapping faces; conversely, when the face is a near object and the potted plant is a distant object, the overlapping area A(2) of the panoramic image is likely to show two overlapping potted plants and a face with a missing corner. This defect is hereinafter referred to as the "overlap / missing corner (OL) phenomenon." Therefore, the industry urgently needs an image processing system and method to improve the image quality under the aforementioned OL phenomenon. Summary of the Invention

[0004] In view of the above problems, one of the objectives of the present invention is to provide an image processing system that can avoid the above-mentioned OL phenomenon and improve image quality when a ROI overlaps with at least one measurement area of ​​an overlapping region.

[0005] According to an embodiment of the present invention, an image processing system is provided, comprising: a multi-lens camera, a vertex list generator, and an image processing apparatus. The multi-lens camera is used to capture a field of view covering an X-degree horizontal field of view and a Y-degree vertical field of view to generate multiple lens images, wherein X <= 360° and Y <= 180°. The vertex list generator is used to perform a set of first operations, including: (a) generating a first principal vertex list according to a correspondence table; and (b) generating a first ROI vertex list according to the first principal vertex list and a positional information of the ROI when a region of interest (ROI) overlaps with at least one measurement area within an overlapping area of ​​a projected image. The image processing apparatus, in a display mode, is used to generate the projected image based on the multiple lens images and a second principal vertex list associated with the first principal vertex list. The first primary vertex list contains a plurality of first vertices, and the plurality of first vertices form a polygonal mesh for modeling the projected image; and the first ROI vertex list contains a plurality of second vertices, and the plurality of second vertices form a cross section relating to the intersection of the ROI and the at least one measurement region.

[0006] Another embodiment of the present invention provides an image processing method applicable to an image processing system. The system includes a multi-lens camera for capturing a field of view covering an X-degree horizontal field of view and a Y-degree vertical field of view to generate multiple lens images. The method includes: obtaining a first principal vertex list according to a correspondence table; when a region of interest (ROI) overlaps with at least one measurement region within an overlap area of ​​a projected image, obtaining a first ROI vertex list according to the first principal vertex list and a positional information of the ROI; and generating the projected image according to the multiple lens images and a second principal vertex list; wherein the second principal vertex list is related to the first principal vertex list; wherein the first principal vertex list includes a plurality of first vertices, and the plurality of first vertices form a polygonal mesh for modeling the projected image; wherein the first ROI vertex list includes a plurality of second vertices, and the plurality of second vertices form a cross section relating to the intersection of the ROI and the at least one measurement region; and wherein X <= 360, Y <= 180.

[0007] The above and other objects and advantages of the present invention will be described in detail below with reference to the following illustrations, detailed descriptions of the embodiments, and the scope of protection of the claims. Attached Figure Description

[0008] Figure 1A This illustrates the relationship between a cube structure and a sphere.

[0009] Figure 1B Displaying an equidistant rectangular panoramic image, derived from the equidistant rectangular projection of four lens images (back, left, right, and front) of an image capturing module 21, and the aforementioned OL phenomenon generated by the overlapping area A(2).

[0010] Figure 2 According to an embodiment of the present invention, a block diagram of a projection image processing system is shown.

[0011] Figure 3A Display a triangular mesh to model the surface of a sphere.

[0012] Figure 3B Display a polygonal grid to compose / model the isometric rectangular panoramic image.

[0013] Figure 4A An example of a grid is used to model an isometric rectangular panoramic image.

[0014] Figure 4B and Figure 4A For example, an equidistant rectangular panoramic image contains twenty control points R(1) to R(20) and twenty measurement areas M(1) to M(20), and each measurement column contains 5 control points (P1 = 5).

[0015] Figure 4C The vertex processing device 210 improves the image defects caused by the mismatch caused by the offset ofs (the distance between the lens center 46 and its camera system center 43) by modifying the texture coordinates of all vertices in each lens image according to the optimal bonding coefficients C(1) to C(20).

[0016] Figure 5A Example: In an isometric rectangular panoramic image 500, two detected ROIs 510 and 520 overlap with two overlapping regions A(2) and A(4).

[0017] Figure 5B The positional relationship between two vertices c2 and P and ten control points R(1) to R(10) in the left-side camera image is illustrated.

[0018] Figure 5C This example illustrates four positional relationships between a detected ROI and a quadrilateral (with four vertices V0 to V3) in a measurement column of an overlapping region.

[0019] Figure 6A According to one embodiment of the present invention, a flowchart of a method for determining the optimal joining coefficient of control points is shown (executed by the splicing decision unit 230 in measurement mode).

[0020] Figures 6B-6C According to an embodiment of the present invention, a flowchart of a method for making coefficient decisions in step S612 is shown (executed by the splicing decision unit 230).

[0021] Figure 6D Example in a connection meter.

[0022] Figure 6E According to an embodiment of the present invention, a flowchart of a method for merging temporary bonding coefficients in step S616 is shown (executed by the splicing decision unit 230).

[0023] Figure 7 A schematic diagram of the image processing apparatus is shown according to an embodiment of the present invention.

[0024] Icon labels:

[0025] 11. Cube Architecture

[0026] 12 spheres

[0027] 15 Non-overlapping regions

[0028] 21 Image Capture Module

[0029] 22 Compensation device

[0030] 23 Correspondence Generator

[0031] 25 Artificial Intelligence (AI) Module

[0032] 40 Ideal Imaging Position

[0033] 41, 44 Imaging Center

[0034] 42. Location of the control area

[0035] 43 Image Capture Module 21 System Center

[0036] 45 objects

[0037] 46, 47 Center of the lens

[0038] 48 Actual imaging position

[0039] 54. Center of the image from the left-hand camera.

[0040] 71 Rasterization Engine

[0041] 72 Texture Mapping Circuit

[0042] 721 and 722 texture mapping engines

[0043] 73 Hybrid Unit

[0044] 74. Target Buffer

[0045] 200 Projection Image Processing System

[0046] 210 Vertex Processing Unit

[0047] 220 Image Processing Device

[0048] 230 splicing decision units

[0049] 240 Vertex List Generator

[0050] θ1 First angle

[0051] θ2 Second angle

[0052] V1 First Vector

[0053] V2 Second Vector

[0054] V3 Third Vector Detailed Implementation

[0055] Throughout this specification and in subsequent claims, the singular forms of terms such as "a" and "the" have both singular and plural meanings, unless otherwise specified herein. The definitions of related terms used throughout this specification and in subsequent claims are as follows, unless otherwise specified herein. Throughout this specification, circuit elements with the same function use the same reference numerals.

[0056] One of the features of this invention is that when a cross-section (related to the ROI detected by AI module 25, such as...) Figure 5A When a cross section overlaps with or occupies more than half of the area of ​​a second measurement area (adjacent to the first measurement area), the temporary warping coefficient of the cross section is used instead of the temporary warping coefficient of a selected control point located in the first measurement area. Accordingly, the present invention avoids the aforementioned OL phenomenon and effectively improves image quality.

[0057] Figure 2 According to an embodiment of the present invention, a block diagram of a projection image processing system is shown. The projection image processing system 200 of the present invention includes an image capturing module 21, a compensation device 22, an artificial intelligence (AI) module 25, and a correspondence generator 23. The compensation device 22 includes a vertex processing device 210, an image processing device 220, a stitching decision unit 230, and a vertex list generator 240. The compensation device 22 receives a correspondence table from the correspondence generator 23 and multiple lens images from the image capturing module 21 to generate a projection image, such as a panoramic image or a wide-angle image.

[0058] The image capturing module 21 is a multi-lens camera capable of simultaneously capturing a field of view encompassing an X-degree horizontal FOV and a Y-degree vertical FOV to generate multiple lens images, where X <= 360 and Y <= 180. For example, such as Figure 1A As shown, the image capturing module 21 includes four lenses (not shown) mounted on the four working surfaces of a cubic structure 11 to simultaneously capture a field of view with a 360-degree horizontal field of view (FOV) and a 90-degree vertical field of view (FOV), thereby generating four-lens images. Note that the invention does not limit the number of lenses included in the image capturing module 21 as long as a field of view with an X-degree horizontal FOV and a Y-degree vertical FOV can be captured, where X <= 360 and Y <= 180. A necessary condition for mounting the image capturing module 21 is that there should be sufficient overlap between the fields of view of any two adjacent lenses to facilitate image stitching.

[0059] The terms used throughout this specification and in subsequent requests are defined as follows, unless otherwise specified in this specification. The term “texture coordinates” refers to coordinates in a texture space (such as a texture image or a lens image); the term “destination space coordinates” refers to coordinates in a destination space (such as a projected image); the term “rasterization operation” refers to the calculation of texture coordinates that map scene geometry (or a projected image) to the various lens images.

[0060] Many projection methods are applicable to the projection image processing system 200 of this invention. The term "projection" refers to flattening the surface of a sphere into a two-dimensional plane, such as a projection plane. This projection includes, but is not limited to, equidistant rectangular projection, cylindrical projection, and modified cylindrical projection. Modified cylindrical projection includes, but is not limited to, Miller projection, Mercator projection, Lambert cylindrical equalarea projection, Pannini projection, etc. Accordingly, the aforementioned projected image includes, but is not limited to, an equidistant rectangular projection image, a cylindrical projection image, and a modified cylindrical projection image. Figures 1A-1B , Figures 3A-3B , Figures 4A-4C and Figure 5ARegarding isometric rectangular projection. Implementations of cylindrical projection and modified cylindrical projection are well known to those skilled in the art and will not be described in detail here. Note that regardless of the projection method used by the projection image processing system 200 of this invention, the correspondence generator 23 will generate a corresponding correspondence table (described below), which defines the vertex mapping relationship between the plurality of lens images and the projected image. For clarity and convenience, the following examples and embodiments are illustrated using isometric rectangular projection, assuming that the image capturing module 21 is a four-lens camera and the projected image is an isometric rectangular panoramic image. It should be noted that the operation mode and image processing method of the projection image processing system 200 of this invention ( Figures 6A-6E It also applies to wide-angle cameras, cylindrical projections, and corrected cylindrical projections.

[0061] The projection image processing system 200's processing pipeline is divided into an offline phase and a connection phase. In the offline phase, the four lenses of the image capturing module 21 are calibrated. For example, many circles are drawn on the surface of a sphere 12 with a radius of 2 meters (R=2) to represent longitude and latitude, and their multiple intersections are considered as multiple calibration points. The four lenses capture these multiple calibration points, and the positions of these multiple calibration points on the multiple lens images are known. Then, because the view angles of the multiple calibration points and the multiple texture coordinates are connected, a mapping relationship between the isometric rectangular panoramic image and the multiple lens images can be established. In this specification, a calibration point with the above mapping relationship is defined as a "vertex". After calibration, the correspondence generator 23 generates a correspondence table with multiple vertices, each vertex providing a mapping relationship between the isometric rectangular coordinates (longitude and latitude) and the texture coordinates of the multiple lens images. In the offline phase, the correspondence generator 23 performs all necessary calculations.

[0062] According to the correspondence table, vertex list generator 240 uses a suitable image registration technique to generate an original master vertex list (as shown in Table 1), and each vertex in the original master vertex list provides a mapping relationship between the isometric rectangular projection image and the plurality of lens images (or between the isometric rectangular coordinates and the plurality of texture coordinates). The original master vertex list is a list of multiple vertices that form the polygonal mesh. Figure 3B The list contains multiple quadrilaterals and / or triangles, with each vertex defined by a corresponding data structure. This data structure defines the vertex mapping between a destination space and a texture space (or between the isometric rectangular coordinates and the texture coordinates). Table 1 shows an example of the data structure for each vertex in the original master vertex list.

[0063]

[0064]

[0065] Figure 3A Display a triangular mesh to model the surface of a sphere. (Reference) Figure 3A A triangular mesh was used to model the surface of a sphere 12. Figure 3B Display a polygon mesh to compose / model the isometric rectangular projection image. Through... Figure 3A The triangular mesh is generated by performing an equidistant rectangular projection. Figure 3B The polygonal mesh, and Figure 3B A polygonal mesh is a set of multiple quadrilaterals and / or multiple triangles defined by the aforementioned vertices. Figure 4A This example illustrates a grid used to model an isometric rectangular panoramic image. (Reference) Figure 4A The grid can be viewed as a rectangular structure with five rows and sixteen columns within an equidistant rectangular wide-angle image. Figure 4A The grid example corresponds to Figure 3B A polygonal mesh, therefore Figure 4A Vertex / intersection 41 on the mesh corresponds to Figure 3B The vertices of a polygonal mesh. In an equidistant rectangular coordinate system, the horizontal distance between any two adjacent vertices on the horizontal axis is the same, and the vertical distance between any two adjacent vertices on the vertical axis is the same. Figure 4A The horizontal and vertical axes in the figure are respectively represented by θ and θ. To indicate, to represent longitude and latitude. Figure 4A In the example, suppose that the four lenses of the image capturing module 21 simultaneously capture a field of view covering a 160-degree HFOV and a 120-degree VFOV to generate four lens images, thereby forming the equidistant rectangular panoramic image.

[0066] According to the present invention, each overlapping area in a projected image contains a column of P1 control points, where P1>=3. The size of the multiple overlapping areas varies depending on the FOV of the four lenses of the image capturing module 21, the lens sensor resolution, and the lens mounting angle. Generally, the width of each overlapping area in the projected image is greater than or equal to a multiple of the width of a column of quadrilaterals. If each overlapping area contains multiple columns of quadrilaterals, one column of quadrilaterals (hereinafter referred to as the "measurement column") is preset within these multiple columns to accommodate the P1 control points. Figure 4AIn the example, the equidistant rectangular panoramic image contains four overlapping areas A(1) to A(4), and each overlapping area contains only one column of quadrilaterals. Therefore, this column of quadrilaterals is directly used as the measurement column to accommodate P1 control points. In each measurement column, the top and bottom two control points are placed at the center of the top and bottom quadrilaterals, respectively. Then, the remaining (P1-2) control points are evenly distributed between the top and bottom two control points. The range of the measurement area (used to measure the area error in the measurement mode) is defined as follows: For a quadrilateral i in the measurement column of each overlapping area, search for the closest control point j, and classify or define the quadrilateral i as measurement area j (corresponding to control point j), where 1<=i<=n1, 1<=j<=P1, and n1 represent the number of quadrilaterals contained in each measurement column. Figure 4B and Figure 4A For example, an equidistant rectangular panoramic image contains twenty control points R(1) to R(20) and twenty measurement areas M(1) to M(20), and each measurement column contains 5 control points (P1 = 5). Figure 4B In the above, twenty control points R(1) to R(20) each have twenty optimal bonding coefficients C(1) to C(20), and the multiple optimal bonding coefficients C(1) to C(20) respectively represent different bonding degrees of the twenty control points R(1) to R(20).

[0067] During the connection phase, the projection image processing system 200 operates in either a measurement mode or a display mode. In measurement mode, the vertex list generator 240 generates an original primary vertex list based on the aforementioned correspondence table, and generates an original ROI vertex list for at least one cross-section based on the original primary vertex list and ROI information from the AI ​​module 25 (described later); the vertex processing device 210 receives the original primary vertex list, the original ROI vertex list, and the test joining coefficient C from the stitching decision unit 230. t (1)~C t (20) Correct the texture coordinates of each vertex in the original master vertex list and the original ROI vertex list in all lens images in a vertex-by-vertex manner, and generate a corrected master vertex list (as shown in Table 2) and a corrected ROI vertex list; based on the corrected master vertex list, the corrected ROI vertex list and the four lens images from the image capturing module 21, the image processing device 220 measures the area error of all measurement areas M(1) to M(20) and the at least one cross-section in the equidistant rectangular panoramic image, and outputs the plurality of area error values. Then, the stitching decision unit 230 determines the texture coordinates of each vertex in the original master vertex list and the original ROI vertex list in all lens images in all lens images in a vertex-by-vertex manner, and generates a corrected master vertex list (as shown in Table 2) and a corrected ROI vertex list; based on the corrected master vertex list, the corrected ROI vertex list and the four lens images from the image capturing module 21, the image processing device 220 measures the area error of all measurement areas M(1) to M(20) and the at least one cross-section in the equidistant rectangular panoramic image, and outputs the plurality of area error values. After that, the stitching decision unit 230 determines the texture coordinates of each vertex in the original master vertex list and the original ROI vertex list in all lens images in a vertex-by-vertex manner, and generates a corrected master vertex list (as shown in Table 2) and a corrected ROI vertex list; based on the corrected master vertex list, the corrected ROI vertex list and the four lens images from the image capturing module 21, the image processing device 220 measures the area error of each measurement area M(1) to M(20) and the corrected ROI vertex list in all lens images in all lens images in a vertex-by-vertex manner, and generates a corrected master vertex list (as shown Figure 4C Set a set of test bonding coefficients C t (1)~C tThe value of (20) is then used to receive all area error values ​​to generate a 2D master error table and a 2D ROI error table. Based on the 2D master error table and the 2D ROI error table, the optimal bonding coefficients C(1) to C(20) for twenty control points R(1) to R(20) are generated (please refer to...). Figures 6A-6E (See the description). On the other hand, in a display mode, the stitching decision unit 230 and the vertex list generator 240 are disabled. At the same time, the vertex processing device 210 and the image processing device 220 operate together to generate an isometric rectangular panoramic image based on the four lens images output by the image capturing module 21 and the twenty optimal stitching coefficients C(1) to C(20) output by the stitching decision unit 230. The detailed operation of the vertex processing device 210 and the image processing device 220 in the display mode to generate the isometric rectangular panoramic image has been disclosed in Taiwan Patent Nos. I615810, I728620 and I622021, and Taiwan Patent Application No. 110149260 (the contents of the above patents are incorporated herein by reference in their entirety as part of this specification).

[0068] Figure 4C The vertex processing device 210 improves the image defects caused by the mismatch caused by the offset ofs (the distance between the lens center 46 and its camera system center 43) by modifying the texture coordinates of all vertices in each lens image according to the optimal joining coefficients C(1) to C(20). Ideally, the lens centers of the four lenses of the image capturing module 21 are simultaneously located at the camera system center 43 (not shown) of the architecture 11, so the single ideal imaging point 40 of a distant object 45 is located on the image plane 12 with a radius of 2 meters (R=2). Taking lenses #1 and #2 as examples, since the ideal imaging point 40 of the image of lens #1 matches the ideal imaging point 40 of the image of lens #2, the equidistant rectangular panoramic image will present a perfect stitching / blending result after the image stitching / blending operation is completed. However, in reality, the lens centers 46-47 of the images of lens #1 and lens #2 have an offset ofs relative to the system center 43, such as Figure 4C As shown on the left, after the image stitching / blending operation is completed, the equidistant rectangular panoramic image clearly reveals the image defects due to mismatch.

[0069] In display mode, vertex processing device 210 sequentially corrects the texture coordinates of each vertex in an original master vertex list in each lens image according to twenty optimal bonding coefficients C(1) to C(20) to generate a corrected master vertex list; in measurement mode, vertex processing device 210 corrects the texture coordinates of each vertex in an original master vertex list in each lens image according to twenty optimal bonding coefficients C(1) to C(20) to generate a corrected master vertex list. t (1)~Ct (20) The texture coordinates of each vertex in each lens image are sequentially modified in an original master vertex list and an original ROI vertex list to generate a modified master vertex list and a modified ROI vertex list. The detailed operation of the vertex processing device 210 in display mode or measurement mode is disclosed in Taiwan Patent No. 1728620, Taiwan Patent Application Nos. 110149260 and 110121015 (the contents of the above patents are incorporated herein by reference in their entirety as part of this specification). Table 2 shows an example of the data structure of each vertex in the modified master vertex list.

[0070]

[0071]

[0072] The vertex processing device 210, based on the plurality of test / optimal bonding coefficients (C... t (1)~C t After (20) or C(1)~C(20)) corrects all texture coordinates of all vertices from the original master / ROI vertex list, the mismatch image defect caused by the lens center offset of the image capturing module 21 (that is, a distance of ofs between the lens center 46 and the system center 43) can be greatly improved (that is, the actual imaging position 48 will be pushed to the ideal imaging position 40, such as Figure 4C As shown. Note that because sphere 12 is virtual, object 45 may be located outside, inside, or on the surface of sphere 12. Also note that the data structures of each vertex in the original ROI vertex list and the modified ROI vertex list are similar to the data structures of each vertex in the original master vertex list and the modified master vertex list (as shown in Table 1 and Table 2), but exclude (do not include) the stitching and blending weights (w) of the i-th shot image. i )" column.

[0073] After system initialization, the projection image processing system 200 needs to operate in measurement mode and display mode at least once to generate an isometric rectangular panoramic image and transmit it to the AI ​​module 25. The AI ​​module 25 uses a trained model to perform at least one of human detection, face detection, and head detection on the isometric rectangular panoramic image to locate at least one ROI within the isometric rectangular panoramic image and generate information about the at least one ROI (hereinafter referred to as "ROI information"). The ROI information includes, but is not limited to, the isometric rectangular coordinates of multiple vertices of the at least one ROI. The above-mentioned use of a trained model for human detection, face detection, and head detection includes, but is not limited to, the following two methods: (1) Deep learning, used for artificial intelligence detection, such as using the programming language Python with the computer vision library OpenCV, the open-source neural network library Keras, and the open-source database TensorFlow. Image information is extracted by OpenCV for image processing, and the neural network model provided by Keras is used with TensorFlow as the backend for model training; (2) Machine learning, which can be implemented using Hidden Markov Models (HMM) and Support Vector Machines (SVM). Basically, traditional machine learning requires developers to decide what features to use (for example, turning an image of a person into a gradient array), and then pass it to the learning model (such as SVM or HMM) for training and testing. Deep learning, on the other hand, allows the model itself to determine feature points (or rather, feature point extraction is already included within the module). Therefore, the image can be directly passed to a deep learning module (such as a Multilayer Perceptron (MLP), Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), etc.) for training and testing. The techniques of deep learning and machine learning are well-known to those skilled in the art and will not be elaborated upon here. It should also be noted that the aforementioned deep learning and machine learning methods are merely examples and not limitations of this invention; other existing or future methods for human / face / head detection can also be applied to the concepts of this invention.

[0074] The AI ​​module 25 of this invention can be implemented in software, hardware, or a combination of software (or firmware) and hardware. In one embodiment, the AI ​​module 25 is implemented using an AI chip. In one embodiment, the AI ​​chip is implemented using a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a combination thereof, specifically for performing AI computations. The aforementioned GPU is often used for the initial development and improvement of AI algorithms (such as the aforementioned deep learning and machine learning methods), a process commonly referred to as "training." The aforementioned ASIC is used for training or inference. In another embodiment, the AI ​​algorithms (such as the aforementioned deep learning and machine learning methods) are executed by a general-purpose chip, such as a central processing unit (CPU) or a general-purpose processor. In other words, the AI ​​module 25 can be implemented using a general-purpose processor and a program memory, wherein the program memory stores a processor-executable program relating to the aforementioned deep learning and machine learning methods. When the general-purpose processor executes the processor-executable program, the general-purpose processor is configured to operate as the AI ​​module 25.

[0075] Figure 5A Example: In an isometric rectangular panoramic image 500, two detected ROIs 510 and 520 overlap with two overlapping regions A(2) and A(4). Figure 5A In the example, suppose (1) the image processing device 220 generates the isometric rectangular panoramic image 500 and transmits it to the AI ​​module 25; (2) the AI ​​module 25 performs human detection, face detection or head detection on the isometric rectangular panoramic image 500 to locate two ROIs 510 and 520, and transmits the corresponding ROI information (including the isometric rectangular coordinates of multiple vertices of ROIs 510 and 520) to the vertex list generator 240. Based on the original primary vertex list, vertex list generator 240 takes the intersection of the equidistant rectangular coordinates of the four vertices of ROI 510 and the measurement column of the overlapping area A(2) to obtain a cross section 511 and the equidistant rectangular coordinates of its four vertices c0 to c3. It also takes the intersection of the equidistant rectangular coordinates of the four vertices of ROI 520 and the measurement column of the overlapping area A(4) to obtain three cross sections 521 to 523 and the equidistant rectangular coordinates of its eight vertices c4 to c7 and a6 to a9. On the other hand, if the ROI information indicates that the ROI will not overlap with any measurement area of ​​the overlapping areas A(1) to A(4), vertex list generator 240 will not generate any cross sections or any original ROI vertex list.

[0076] Based on the original main vertex list, the equidistant rectangular coordinates of the four vertices c0 to c3, and the equidistant rectangular coordinates of the closest (vertices c0 to c3) vertices a0 to a5 in the measurement column of the overlapping area A(2), the vertex list generator 240 interpolates the texture coordinates of vertices a0 to a5 in each shot image to obtain the corresponding texture coordinates of each vertex c0 to c3 in each shot image. Based on the texture coordinates of each vertex c0 to c3, control points R(6) to R(10), and the image center in each shot image, the vertex list generator 240 determines the "joining coefficient index (idx) of the i-th shot image" in the data structure of each vertex c0 to c3 in the original ROI vertex list. i0 idx i1 The column ")" is used to calculate the blending weight (Alpha) of the blending coefficient in the i-th shot image. i )" column.

[0077] Figure 5B Example: A vertex c2 in the left-hand camera image (e.g.) Figure 5A The positional relationship between the ten control points R(1) to R(10). Figure 5B In the example, the first angle θ1 is clockwise and is formed between a first vector V1 and a second vector V2; the first vector V1 is centered at image center 54 (with texture coordinates (u center ,v center Starting from the image center 54 and ending at the control point R(8), the second vector V2 starts from the image center 54 and ends at the target vertex c2(u). C2 ,v C2 The endpoint is θ2. The second angle θ2 is clockwise and is formed between a third vector V3 and the second vector V2; the third vector V3 starts from the image center 54 and ends at the position of control point R(7). The vertex list generator 240 determines which two control points (i.e. R(7) and R(8)) are closest to the target vertex c2, and writes their indices (7 and 8) into the "Joining Coefficient Index" field of the left-hand lens image of the data structure of the target vertex c2 in the original ROI vertex list; in addition, the vertex list generator 240 also calculates the best / test joining coefficients (C(7) and C(8), or C t (7) and C tThe mixed weight (=θ1 / (θ1+θ2)) of (8) is written into the "Mixed Weight (Alpha) of Joining Coefficient" field of the data structure of the target vertex c2 in the original ROI vertex list. Assuming that the stitching decision unit 230 generates twenty optimal joining coefficients in sequence at the end of the measurement mode, in the display mode, the vertex processing device 210 will use the values ​​in the "Joining Coefficient Index" field and the "Mixed Weight (Alpha) of Joining Coefficient" field to substitute into the following equation to calculate the interpolation joining coefficient C' of the target vertex c2 in the left lens image C' = C(7)×(θ1 / (θ1+θ2)))+C(8)×(θ2 / (θ1+θ2)) (to be explained later). In this way, the vertex list generator 240 calculates the values ​​of the data structure of vertex c2 in all fields of the left lens image. In the same manner, vertex list generator 240 calculates the values ​​of the data structures of vertices c0 to c7 and a6 to a9 in all fields of all shot images. The original ROI vertex list contains the following vertices in sequence (each with its own data structure): {c0, c1, c2, c3, c4, c5, a6, a7, a8, a9, c6, c7}.

[0078] Figure 5C Four examples are given where a detected ROI overlaps with a quadrilateral (with four vertices V0 to V3) in a measurement column of an overlapping region. In example A, no ROI overlaps with the quadrilateral, so no original ROI vertex list needs to be generated. If any of the cases in examples B to D occur, the projection image processing system 200 will enter measurement mode. In example B, an ROI occupies the entire quadrilateral, so the vertex list generator 240 directly copies the data structure of vertices V0 to V3 from the original main vertex list (but excludes the "stitching blending weights"). i The vertex list generator 240 directly copies the data structure of vertices V2 to V3 from the original main vertex list (excluding the "concatenation and blending weights" field) to the original ROI vertex list. In example C, an ROI occupies part of a quadrilateral, so the vertex list generator 240 directly copies the data structure of vertices V2 to V3 from the original main vertex list (but excludes the "concatenation and blending weights" field). i The data structure includes the original ROI vertex list ("field") and the data structure for calculating the original vertices c0 to c3, along with the texture coordinates and stitching coefficient index (idx) of each image i. i0 idx i1 ) and the combined weights of the bonding coefficients (Alpha) i This forms an initial ROI vertex list containing the following four vertices: {V4, V5, V2, V3}. In example D, an ROI occupies a portion of a quadrilateral, so the vertex list generator 240 calculates the texture coordinates and joining coefficient index (idx) of each vertex V4 to V7 in the data structure of each lens image i. i0idx i1 ) and the combined weights of the bonding coefficients (Alpha) i This will form an original list of ROI vertices containing the following four vertices: {V4, V5, V6, V7}.

[0079] One of the features of this invention is a preset number of loops ( Figure 6A The preset number of loops is related to an offset of 'ofs', which is the distance between the lens center 46 of the image capturing module 21 and its system center 43 (see reference). Figure 4C In measurement mode, according to Figure 4C The offset ofs, the splicing decision unit 230 will use twenty test joining coefficients C t (1)~C t (20) Set to different numerical ranges to measure the area error E(1) to E(20) of twenty measurement areas M(1) to M(20) and the area error E(511) and E(521) to E(523) of four cross-sections 511 and 521 to 523, and set the twenty test joint coefficients to the same value in each (or each loop). For example, assuming ofs = 3 cm, the twenty test joint coefficients C t (1)~C t (20) Set to a numerical range of 0.96 to 1.04, if each increment is 0.01, there will be a total of nine measurements (i.e., Figure 6A (max = 9); Assuming ofs = 1 cm, the twenty test bonding coefficients C t (1)~C t (20) Set to a numerical range of 0.99 to 1.00, if each increment is 0.001, there will be a total of ten measurements (i.e., Figure 6A (max = 10). Note that the offset ofs is detected / determined during the offline phase, so the twenty test engagement coefficients C can be predetermined. t (1)~C t The value of (20).

[0080] Figure 6A According to an embodiment of the present invention, a flowchart of a method for determining the optimal joining coefficient of control points is shown (executed by the splicing decision unit 230 in measurement mode). For ease of explanation, it is assumed below that ofs = 3 cm, and... Figure 4B and Figure 5A To illustrate with examples Figure 6A The method for determining the optimal bonding coefficients C(1) to C(20) for twenty control points R(1) to R(20), Figures 6B-6C The method of the execution coefficient decision step (step S612) and Figure 6E The method for performing the temporary joining coefficient merging step (step S616). Here, Q3 represents the number of cross segments; if no cross segments are generated, Q3 is set to 0. TH1 equals (Q3+1) plus the total number of control points in the equidistant rectangular panoramic image; TH2 equals (P1+1). Figure 4B and Figure 5A In this example, during system initialization, Q3 = 4, TH1 = 24, and TH2 = 6. Note that ofs = 3 cm. Figure 4B and Figure 5A This is merely an example and not a limitation of the invention.

[0081] Step S602: Set the loop number Q1 and the test bonding coefficient to new values ​​respectively. In one embodiment, Q1 is set to 1 in the first loop, and Q1 is increased by 1 in each subsequent loop; if ofs = 3 cm, then max = 9, and in the first loop, the plurality of test bonding coefficients Ct(1) to Ct(20) are all set to 0.96 (i.e., Ct(1) = ... = Ct(20) = 0.96), and in subsequent loops, the plurality of test bonding coefficients Ct(1) to Ct(20) are sequentially set to new values. t (1)~C t (20) is set to 0.97,…,1.04.

[0082] Step S604: Clear all regional error quantities E(i) to 0, where i = 1, 2, ..., 20, 511 and 521 to 523.

[0083] Step S605: Based on the test bonding coefficient C t (1)~C t The value of (20), the original principal vertex list, and the original ROI vertex list are used to generate a modified principal vertex list and a modified ROI vertex list. The following uses... Figure 5B Taking a target vertex P as an example, after receiving the original primary vertex list from the vertex list generator 240, the vertex processing device 210, based on the "joint coefficient index" field (e.g., 7 and 8) in the left-side camera image of the data structure of the target vertex P, sequentially selects twenty test joint coefficients C. t (1)~C t Two test joining coefficients (C) are extracted from the one-dimensional test joining coefficient array (from the splicing decision unit 230) of (20). t (7) and C t (8) Then, based on the "blending weight (Alpha)" field of the left-side camera image (see Table 1) in the data structure of the target vertex P (e.g., θ1 / (θ1+θ2)), calculate the interpolation blending coefficient C' according to the following equation: C'=C t(7)×(θ1 / (θ1+θ2))+C t (8)×(θ2 / (θ1+θ2)). Then, the vertex processing device 210 calculates the corrected texture coordinates (u) of the target vertex P in the left-side lens image according to the following equation. P ',v P '):u P '=(u P -u center )×C'+u center ;v P '=(v P -v center )×C'+v center In this manner, the vertex processing device 210 performs the calculations based on the twenty test bonding coefficients C. t (1)~C t (20) The texture coordinates of each vertex from the original master vertex list in the left-side camera image are sequentially corrected to generate a portion of a corrected master vertex list (as shown in Table 2). Similarly, the vertex processing device 210 also adjusts the texture coordinates of each vertex from the original master vertex list in the left-side camera image based on the twenty test bonding coefficients C. t (1)~C t (20) The texture coordinates of each vertex from the original master vertex list in the front, right, and back lens images are corrected sequentially to complete the corrected master vertex list (as shown in Table 2). Similarly, the vertex processing device 210 adjusts the texture coordinates of each vertex from the original master vertex list in the front, right, and back lens images. t (1)~C t (20) Sequentially modify the texture coordinates of each vertex in the data structure of each vertex from the original ROI vertex list in each shot image to generate a modified ROI vertex list. As mentioned above, the data structure of each vertex in the modified ROI vertex list is similar to the data structure of each vertex in the modified main vertex list (as shown in Table 2), but excludes the "stitching and blending weights (w) of the i-th shot image". i )" column.

[0084] Step S606: The image processing device 220, based on the corrected principal vertex list, the corrected ROI vertex list, and the four lens images from the image capturing module 21, measures the area error E(i) of twenty measurement areas M(1) to M(20) and four cross-sections 511 and 521 to 523 of the equidistant rectangular wide-angle image, where i = 1, ..., 20, 511, and 521 to 523 (to be measured in the image processing device 220). Figure 7 illustrate).

[0085] Step S608: Store all area error quantities in a two-dimensional (2D) master error table and a 2D ROI error table. Table 3A shows an example of the 2D master error table when ofs = 3 cm (the test bonding coefficient ranges from 0.96 to 1.04). In the 2D master error table of Table 3A, there are a total of nine different values ​​for the test bonding coefficient (0.96 to 1.04), each value corresponding to a set of twenty area error quantities E(1) to E(20).

[0086]

[0087] Table 3B shows an example of the 2D ROI error table when ofs = 3 cm (the test bonding coefficient ranges from 0.96 to 1.04). In the 2D ROI error table in Table 3B, there are a total of nine different test bonding coefficient values ​​(0.96 to 1.04), each value corresponding to a set of four area error quantities E(511) and E(521) to E(523).

[0088]

[0089] Step S610: Determine whether the loop count Q1 has reached the maximum value of 9. If yes, skip to step S612; otherwise, return to step S602.

[0090] Step S612: Based on the above 2D master error table and 2D ROI error table, perform coefficient decision operation to obtain twenty control points R(1)~R(20) and twenty-four temporary bonding coefficients (C) for four cross segments 511 and 521~523. P (1)~C P (20) C P (511) and C P (521)~C P (523)).

[0091] Step S614: Determine whether Q3 is greater than 0. If yes, it means that at least one cross segment has been generated, and jump to step S616; otherwise, it means that no cross segment has been generated, and jump to step S618.

[0092] Step S616: Temporary joint coefficients (C) of the four transverse segments 511 and 521-523 P (511) and C P (521)~C P (523)) The temporary bonding coefficients are combined into the temporary bonding coefficients C of the twenty control points R(1)~R(20). P (1)~C P (20).

[0093] Step S618: Output temporary bonding coefficient C P (j) is taken as the optimal joining coefficient C(j), where j = 1, 2, ..., 20. In this display mode, the plurality of optimal joining coefficients C(i) are output to the vertex processing device 210 to generate a corresponding corrected master vertex list, so that the image processing device 220 generates a corresponding equidistant rectangular panoramic image based on the corresponding corrected master vertex list and the four lens images from the image capturing module 21.

[0094] Figures 6B-6C This is a flowchart illustrating the coefficient decision operation in step S612 according to an embodiment of the present invention (executed by the splicing decision unit 230). Please refer to the following... Figures 6B-6C This describes all the steps involved in performing the coefficient decision operation.

[0095] Step S632: Set Q2 to 1 for initialization.

[0096] Step S634: Determine if Q2 is greater than 20. If yes, skip to step S638 to process a cross-section; otherwise, skip to step S636 to process a control point.

[0097] Step S636: Extract information of a selected control point from the above 2D main error table (as shown in Table 3A). For example, if Q2 = 10, extract information of a selected control point R(10) from the above 2D main error table (as shown in Table 3A). This information includes all the above test bonding coefficients and the nine area error quantities corresponding to the selected control point.

[0098] Step S638: Extract information for a selected cross-section from the 2D ROI error table (as shown in Table 3B). For example, if Q2 = 21, extract information for a selected cross-section 511 from the 2D ROI error table (as shown in Table 3B); if Q2 = 22, extract information for a selected cross-section 521 from the 2D ROI error table (as shown in Table 3B); ...; if Q2 = 24, extract information for a selected cross-section 523 from the 2D ROI error table (as shown in Table 3B); this information includes all the test bonding coefficients and the error amounts of the nine regions corresponding to the selected cross-section.

[0099] Step S640: Determine the local minimum value among the area error quantities of the selected control point or cross section (511 and 521-523). Table 4A shows an example of nine values ​​of the area error quantities E(6)-E(8) of the measurement area M(6)-M(8) and the test joint coefficient.

[0100]

[0101]

[0102] As shown in Table 4A, there is only one local minimum among the nine regional error quantities in E(6), while there are two local minimums among the nine regional error quantities in E(7) and E(8), with each local minimum marked with an asterisk (*).

[0103] Table 4B shows an example of the area error E(511) of the cross section 511 and nine values ​​of the test joint coefficient. Two of the nine area error values ​​of E(511) are local minima.

[0104]

[0105] Step S642: Based on the plurality of local minima of the selected control point or cross section (511 and 521-523), candidates are selected and stored. Table 5 shows the candidates selected from the plurality of local minima in Table 4A. The number of candidates is equal to the number of local minima in Table 4A. Taking the selected control point R(7) (i.e., Q2=7) as an example, the selected control point R(7) has two candidates [0] and [1], each candidate containing a corresponding index (ID), a corresponding test bonding coefficient (WC), and a corresponding area error (RE). The two candidates of the selected control point R(7) are stored in a memory device or a buffer (not shown).

[0106]

[0107] Please note that in steps S640 to S642, the operation method for each selected cross-section (511, 521 to 523) is the same as the operation method for each selected control point, so it will not be described again here. Here, the value of the loop count Q2 is incremented by 1.

[0108] Step S644: Determine if Q2 is equal to TH1 (=25). If yes, skip to step S646; otherwise, skip to step S634 to process the next control point or cross section.

[0109] Step S646: Set Q2 to 1 for initialization.

[0110] Step S648: Based on a selected control point / cross section, determine a corresponding decision group to extract all candidates for control points / cross sections within the corresponding decision group.

[0111] Back Figure 4BTypically, each control point is adjacent to two other control points. A selected control point and its two adjacent control points form a selected decision group to determine the optimal bonding coefficient of the selected control point. For example, a selected control point R(9) and its two adjacent control points R(8) and R(10) form a selected decision group. However, if a selected control point (such as R(6)) is located at the top or bottom of the overlap area A(2), then the selected control point R(6) will only form a selected decision group with its only adjacent control point R(7) to determine its temporary bonding coefficient C. t (6). Therefore, according to Table 3A and Figure 4B There are a total of twenty decision groups associated with control points. Based on the 2D ROI error table in Table 3B and... Figure 5A Only the following four decision groups are related to the cross section: the first contains only cross section 511; the second contains cross section 521 and its only adjacent cross section 522; the third contains cross section 522 and two adjacent cross sections 521 and 523; and the fourth contains cross section 523 and its only adjacent cross section 522.

[0112] Accordingly, the memory device or buffer mentioned in step S642 simultaneously retrieves candidates for all control points (i.e., a selected control point and one or two adjacent control points) in the same decision group; or, the memory device or buffer mentioned in step S642 simultaneously retrieves candidates for all cross segments (i.e., a selected cross segment and zero or one or two adjacent cross segments) in the same decision group. For example, if Q2 = 11, then candidates for a selected control point R(11) and its adjacent control point R(12) in the same decision group are simultaneously selected; if Q2 = 21, then candidates for a selected cross section 511 are selected; if Q2 = 22, then candidates for a selected cross section 521 and its adjacent cross section 522 in the same decision group are simultaneously selected; if Q2 = 23, then candidates for a selected cross section 522 and its adjacent cross sections 521 and 523 in the same decision group are simultaneously selected; if Q2 = 24, then candidates for a selected cross section 523 and its adjacent cross section 522 in the same decision group are simultaneously selected. For ease of explanation, the following steps assume a selected control point R(7), and R(7) and its two adjacent control points R(6) and R(8) form a selected decision group to determine its temporary bonding coefficient C. P (7).

[0113] Step S649: Determine whether Q2 is greater than 20 and whether the cross-section forms a decision group on its own. If yes, skip to step S656; otherwise, skip to step S650.

[0114] Step S650: Establish a link metric based on all candidates for control points or cross-sections within the same decision group. For example... Figure 6D As shown, a connection measurement is established based on the candidates of control points R(6) to R(8) in Table 5.

[0115] Step S652: Determine the minimum sum of the connection meter values ​​across all paths of the connection meter. (Regarding the two connection meter values...) and The minimum value between the two Regarding the two connection metrics and The minimum value between the two Next, the sum of the connection metrics for path 0-0-0 and path 0-1-1 is calculated as follows: and because Therefore, it can be decided (Path 0-1-1) is the minimum sum of connection meter values ​​across all paths of this connection meter, such as... Figure 6D The solid line path.

[0116] Step S654: Determine the temporary joining coefficient of the selected control point or cross section based on the minimum sum of the above connection measurement values. Regarding the example given in step S652, because... (Path 0-1-1) is the minimum sum of the connected measurement values ​​among all paths, so 1.02 is determined as the optimal bonding coefficient for control point R(7). However, if the sum of the connected measurement values ​​of two or more paths is the same at the end of the calculation, the test bonding coefficient of the node with the smallest area error is selected as the temporary bonding coefficient for the selected control area or cross section. Note that in steps S650 to S654, the operation method for each selected cross section (511, 521 to 523) is the same as the operation method for each selected control point, so it will not be described again here. Here, the value of the loop count Q2 is incremented by 1.

[0117] Step S656: Determine the temporary binding coefficient of the selected cross section based on the global minimum of the area error of the selected cross section. As mentioned above, cross section 511 forms a decision group on its own. In this specific case, referring to Table 4B, since the test binding coefficient 1.00 corresponds to the global minimum of the area error E (511) (i.e., 3000), the test binding coefficient 1.00 is directly selected as the temporary binding coefficient C of cross section 511. P (511). Here, the value of the loop count Q2 is incremented by 1.

[0118] Step S658: Determine whether the loop count Q2 has reached the upper limit TH1 (=25). If yes, end this process; otherwise, return to step S648 to process the next control point or cross section.

[0119] Figure 6E According to an embodiment of the present invention, a flowchart of the method for merging temporary bonding coefficients in step S616 is shown (executed by the splicing decision unit 230).

[0120] Step S670: Set r to 0 for initialization. The parameter r represents the overlap index.

[0121] Step S672: Determine whether the measurement column of the overlapping region A(r+1) overlaps with an ROI. If yes, skip to step S674; otherwise, skip to step S686. Figure 5A In the example, the measurement columns in the overlapping regions A(2) and A(4) overlap with ROI510 and 520, respectively.

[0122] Step S674: Set s to 1 for initialization. The parameter s represents the index of the measurement area in the same column.

[0123] Step S676: Determine whether the measurement area M(r×5+s) overlaps with a cross-section (e.g., 511, 521~523). If yes, skip to step S678; otherwise, skip to step S680.

[0124] Step S678: Replace the temporary bonding coefficient C of the measurement area M(r×5+s) with a temporary bonding coefficient corresponding to the transverse segment. P (r×5+s). For example, in Figure 5A In the example, the temporary bonding coefficient C of the transverse segment 511 P (511), respectively replacing the temporary bonding coefficients C of the measurement areas M(7) and M(8). P (7) and C P (8).

[0125] Step S680: Determine whether the measurement area M(r×5+s) is adjacent to a transverse segment. If yes, skip to step S681; otherwise, skip to step S682. Figure 5A In the example, the cross section 511 overlaps with the measurement areas M(7) and M(8) and is adjacent to the measurement areas M(6) and M(9).

[0126] Step S681: Determine whether the transverse segment occupies more than half of the area of ​​the adjacent measurement area (M(r×5+s-1) or M(r×5+s+1)), where (s-1)>=1 and (s+1)<=5. If yes, skip to step S678 to perform the replacement; otherwise, do not replace and skip to step S682. For example, when r=s=1 (i.e., determine the measurement area M(6)), it is determined that the transverse segment 511 occupies more than half of the area of ​​its adjacent measurement area (M(7)), so in step S678, the temporary bonding coefficient C of the transverse segment 511 must be used. P (511) The temporary bonding coefficient C replacing the measurement area M(6) P (6); When r = 1 and s = 4 (i.e., when judging the measurement area M(9)), it is determined that the cross section 511 occupies less than half of the area of ​​the adjacent measurement area (M(7)), so it is not necessary to use the temporary bonding coefficient C. P (511) The temporary bonding coefficient C replacing the measurement area M(9) P (9).

[0127] Step S682: Increment the value of s by 1.

[0128] Step S684: Determine whether s has reached TH2 (=6). If yes, skip to step S686; otherwise, return to step S676 to process the next measurement area.

[0129] Step S686: Increment the value of r by 1.

[0130] Step S688: Determine if r has reached the upper limit of 4. If yes, end this process; otherwise, return to step S672 to process the next measurement column (or overlapping area).

[0131] Please note that in the examples and embodiments described above, the transverse segment 511 is treated as a single block. In another embodiment, the transverse segment 511 is divided into a first segment (c0, c1, a2, a3) and a second segment (a2, a3, c2, c3); Figures 6A-6E The method for processing the first segment and the second segment is similar to the method for processing the transverse segments 521 to 523, so it will not be described in detail here.

[0132] The method for measuring the area error amounts E(1) to E(20), E(511), and E(521) to E(523) in step S606 is explained below. The following explanation uses the measurement area error amount E(511) as an example. Figure 7 A schematic diagram of the image processing apparatus is shown according to an embodiment of the present invention. Please refer to... Figure 7The image processing apparatus 220 includes a rasterization engine 71, a texture mapping circuit 72, a mixing unit 73, a destination buffer 74, and a measurement unit 75. The texture mapping circuit 72 includes two texture mapping engines 721 to 722.

[0133] In measurement mode, the mixing unit 73 is disabled, and the rasterization engine 71 receives four vertices c0 to c3 from a modified ROI vertex list and enables the control signal CS to enable the measurement unit 75. These four vertices (c0, c1, c2, c3) each contain the following data structures: Vertex c0: {(x c0 ,y c0 ),2,ID Left ,(u 1c0 ,v 1c0 ), ID Front ,(u 2c0 ,v 2c0 )};Vertex c1:{(x c1 ,y c1 ),2,ID Left ,(u 1c1 ,v 1c1 ), ID Front ,(u 2c1 ,v 2c1 )};Vertex c2:{(x c2 ,y c2 ),2,ID Left ,(u 1c2 ,v 1c2 ), ID Front ,(u 2c2 ,v 2c2 )};Vertex c3:{(x c3 ,y c3 ),2,ID Left ,(u 1c3 ,v 1c3 ), ID Front ,(u 2c3 ,v 2c3 )}.

[0134] The rasterization engine 71 uses the following steps to perform a quadrilateral rasterization operation on a point Q (with equidistant rectangular coordinates (x,y)) within quadrilaterals c0 to c3, so as to obtain two texture coordinates on the left and frontal lens images in parallel: (1) Using a bilinear interpolation method, based on the equidistant rectangular coordinates (x,y) c0 ,y c0 ,x c1 ,y c1 ,xc2 ,y c2 ,x c3 ,y c3 (2) Calculate the four spatial weights (a,b,c,d) in the left-side lens image, where a+b+c+d=1; (3) Calculate the sampling point Q in the left-side lens image. L Texture coordinates (corresponding to point Q): (u1, v1) = (a*u 1c0 +b*u 1c1 +c*u 1c2 +d*u 1c3 ,a*v 1c0 +b*v 1c1 +c*v 1c2 +d*v 1c3 ); Calculate the sampling point Q in the frontal lens image. F Texture coordinates (corresponding to point Q): (u2, v2) = (a*u 2c0 +b*u 2c1 +c*u 2c2 +d*u 2c3 ,a*v 2c0 +b*v 2c1 +c*v 2c2 +d*v 2c3Based on the two texture coordinates (u1, v1) and (u2, v2) mentioned above, the two texture mapping engines 721-722 use any suitable method (e.g., nearest-neighbour interpolation, bilinear interpolation, or trilinear interpolation) to texture map the texture data of the left and frontal lens images to generate two sample values ​​s1 and s2. Based on the equidistant rectangular coordinates (x, y) of point Q, the measurement unit 75 can determine whether point Q falls within the cross section 511. If so, the measurement unit 75 can use any known algorithm, such as sum of absolute differences (SAD), sum of squared differences (SSD), median absolute deviation (MAD), etc., to measure the regional error E(511) of the cross section 511. For example, the measurement unit 75 uses the following equations: E=|s1-s2|;E(511)+=E to accumulate the absolute value of the difference between the sampled values ​​of each point in the cross-section 511 in the left lens image and the corresponding point in the cross-section 511 in the front lens image, so as to obtain a SAD value as the area error E(511) of the cross-section 511. In this way, the measurement unit 75 measures the area error E(511) and E(521)~E(523) of the cross-sections 511 and 521~523, as well as the area error E(1)~E(20).

[0135] The compensation device 22 and correspondence generator 23 of the present invention can be implemented in software, hardware, or a combination of software (or firmware) and hardware. An example of a simple solution is a field-programmable gate array (FPGA) or an ASIC. In a preferred embodiment, the vertex processing device 210 and the image processing device 220 are implemented using a graphics processing unit (GPU) and a first program memory; the stitching decision unit 230 and the correspondence generator 23 are implemented using a first general-purpose processor and a second program memory. The first program memory stores a first processor-executable program, and the second program memory stores a second processor-executable program. When the graphics processing unit executes the first processor-executable program, the graphics processing unit is configured to operate as the vertex processing device 210 and the image processing device 220. When the first general-purpose processor executes the second processor-executable program, the first general-purpose processor is configured to operate as the stitching decision unit 230 and the correspondence generator 23.

[0136] In another embodiment, the compensation device 22 and the correspondence generator 23 are implemented using a second general-purpose processor and a third program memory. The third program memory stores a third processor-executable program. When the second general-purpose processor executes the third processor-executable program, the second general-purpose processor is configured to operate as the compensation device 22 and the correspondence generator 23.

[0137] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the patent application of the present invention; any equivalent changes or modifications made without departing from the spirit disclosed in the present invention should be included within the scope of protection of the claims of this application.

Claims

1. An image processing system, characterized in that, Include: A multi-lens camera is used to capture a field of view that covers an X-degree horizontal field of view and a Y-degree vertical field of view to produce multiple lens images, where X<=360 and Y<=180. A vertex list generator used to perform multiple first operations, including: (a) Generate a first primary vertex list based on a correspondence table; and (b) When a region of interest overlaps with at least one measurement region within an overlapping area of ​​a projected image, a first region of interest vertex list is generated based on the first principal vertex list and a position information of the region of interest. An image processing apparatus, in a display mode, is used to generate the projected image based on the plurality of lens images and a second principal vertex list; and A splicing decision-making device is used to perform multiple secondary operations, including: Based on a set of test bonding coefficients and the cumulative pixel value error between all measurement areas and a cross section, a set of optimal bonding coefficients for the control points of all measurement areas is generated. The second main vertex list is related to the first main vertex list; The first primary vertex list contains multiple first vertices, and the multiple first vertices form a polygonal mesh to model the projected image. The first region of interest vertex list contains multiple second vertices, and the multiple second vertices form the cross section, which is related to the intersection of the region of interest and the at least one measurement region. The projected image includes at least one overlapping region, and each overlapping region includes multiple measurement regions; and The second primary vertex list is also related to the optimal bonding coefficient of the group.

2. The system according to claim 1, characterized in that, Also includes: An artificial intelligence module uses a trained model to perform at least one of human detection, face detection, and head detection on the projected image to generate location information of the region of interest.

3. The system according to claim 2, characterized in that, The artificial intelligence module is implemented using an artificial intelligence chip, and the artificial intelligence chip is implemented using a graphics processing unit, an application-specific integrated circuit, or a combination thereof.

4. The system according to claim 1, characterized in that, Also includes: A correspondence generator is used to correct the plurality of first vertices to generate the correspondence table, wherein the correspondence table contains the plurality of first vertices, and each first vertex provides a mapping between the projected image and the plurality of lens images.

5. The system according to claim 1, characterized in that, The first operation (b) includes: Based on the first target coordinates of the multiple first vertices near the multiple second vertices and the second target coordinates of the multiple second vertices, the first texture coordinates of the multiple first vertices in each lens image are interpolated to generate the second texture coordinates of each second vertex in each lens image; Each first vertex has a first data structure and each second vertex has a second data structure. Each first data structure includes a first target coordinate of the first vertex in the projected image and a first texture coordinate of the first vertex in each lens image. Each second data structure includes a second target coordinate of the second vertex in the projected image and a second texture coordinate of the second vertex in each lens image.

6. The system according to claim 5, characterized in that, The first operation (b) further includes: Based on the second texture coordinates of each second vertex in each shot image, calculate the index value of the two nearest control points of each second vertex in each shot image and a coefficient mixing weight to generate the first region of interest vertex list. Each of the second data structures also includes the index values ​​of the corresponding second vertex to the two nearest control points in each lens image, as well as the coefficient mixing weights.

7. The system according to claim 1, characterized in that, Also includes: A vertex processing device, in a measurement mode, generates a third primary vertex list based on a first primary vertex list and the group test merging coefficients, and generates a second region of interest vertex list based on a first region of interest vertex list and the group test merging coefficients.

8. The system according to claim 7, characterized in that, In the measurement mode, the image processing device also measures the cumulative pixel value error between all measurement areas and the cross section based on the plurality of lens images, the third principal vertex list, and the second region of interest vertex list.

9. The system according to claim 1, characterized in that, The plurality of second operations further include: Based on a selected control point, a selected decision group is determined; and The temporary bonding coefficient of the selected control point is determined based on the group test bonding coefficient and the local minimum of the cumulative pixel value error between each control point in the selected decision group, wherein the selected decision group includes the selected control point and one or two neighboring control points.

10. The system according to claim 9, characterized in that, The second operation of generating the optimal cohesion coefficient for the group of control points in all measurement areas further includes: When the transverse segment is a single block The test bonding coefficient corresponding to the overall minimum value of the cumulative pixel value error of the cross section is defined as the temporary bonding coefficient of the cross section; and When the cross section overlaps with a first measurement area or occupies more than half of the area of ​​a second measurement area, the temporary bonding coefficient of the control point located in the first measurement area is replaced by the temporary bonding coefficient of the cross section. The second measurement area is adjacent to the first measurement area.

11. The system according to claim 9, characterized in that, The second operation of generating the optimal cohesion coefficient for the group of control points in all measurement areas further includes: When the transverse segment is divided into multiple segments and the multiple segments overlap with multiple measurement areas Based on a selected segment, a selected decision group is determined; as well as The temporary bonding coefficient of the selected segment is determined based on the group test bonding coefficient and the local minimum of the cumulative pixel value error among the segments in the selected decision group, wherein the selected decision group includes the selected segment and one or two neighboring segments.

12. The system according to claim 11, characterized in that, The second operation of generating the optimal cohesion coefficient for the group of control points in all measurement areas further includes: When the transverse segment is divided into multiple segments and the multiple segments overlap with multiple measurement areas After the second operation of determining the temporary engagement coefficient of the selected segment, when one of the plurality of segments overlaps with a first measurement area or occupies more than half of the area of ​​a second measurement area, the temporary engagement coefficient of the control point located in the first measurement area is replaced by the temporary engagement coefficient of the one of the plurality of segments. The second measurement area is adjacent to the first measurement area.

13. The system according to claim 1, characterized in that, The set of test coupling coefficients is related to the offset of the center of one lens of the multi-lens camera relative to the center of the camera system.

14. An image processing method, characterized in that, A method applicable to an image processing system comprising a multi-lens camera for capturing a field of view encompassing an X-degree horizontal field of view and a Y-degree vertical field of view to generate multiple-lens images, the method comprising: Based on a correspondence table, a list of first principal vertices is obtained; When a region of interest overlaps with at least one measurement region within an overlapping area of ​​a projected image, a first region of interest vertex list is obtained based on the first principal vertex list and a positional information of the region of interest. The projected image is generated based on the multiple camera images and a second primary vertex list; as well as Based on a set of test bonding coefficients and the cumulative pixel value error between all measurement areas and a cross section, a set of optimal bonding coefficients for the control points of all measurement areas is obtained. The second main vertex list is related to the first main vertex list; The first primary vertex list contains multiple first vertices, and the multiple first vertices form a polygonal mesh to model the projected image. The first region of interest vertex list contains multiple second vertices, and the multiple second vertices form the cross section, which is related to the intersection of the region of interest and the at least one measurement region. Where X<=360, Y<=180; The projected image includes at least one overlapping region, and each overlapping region includes multiple measurement regions; and The second primary vertex list is also related to the optimal bonding coefficient of the group.

15. The method according to claim 14, characterized in that, Also includes: Before obtaining the vertex list of the first region of interest, a trained model is used to perform at least one of human detection, face detection, and head detection on the projected image to generate the location information of the region of interest.

16. The method according to claim 14, characterized in that, Also includes: Before obtaining the first primary vertex list, the plurality of first vertices are corrected to generate the correspondence table, wherein the correspondence table contains the plurality of first vertices, and each first vertex provides a mapping between the projected image and the plurality of lens images.

17. The method according to claim 14, characterized in that, The step of obtaining the vertex list of the first region of interest further includes: Based on the first target coordinates of the multiple first vertices near the multiple second vertices and the second target coordinates of the multiple second vertices, the first texture coordinates of the multiple first vertices in each lens image are interpolated to generate the second texture coordinates of each second vertex in each lens image; Each first vertex has a first data structure and each second vertex has a second data structure. Each first data structure includes a first target coordinate of the first vertex in the projected image and a first texture coordinate of the first vertex in each lens image. Each second data structure includes a second target coordinate of the second vertex in the projected image and a second texture coordinate of the second vertex in each lens image.

18. The method according to claim 17, characterized in that, The step of obtaining the vertex list of the first region of interest further includes: Based on the second texture coordinates of each second vertex in each lens image, calculate the index values ​​of the two nearest control points of each second vertex in each lens image and a coefficient mixing weight to generate the first region of interest vertex list. Each of the second data structures also includes the index values ​​of the two nearest control points of the corresponding second vertex in each lens image and the coefficient mixing weights.

19. The method according to claim 14, characterized in that, The optimal cohesion coefficient for obtaining the control points of all measurement areas includes: Based on the first primary vertex list and the group test bonding coefficient, a third primary vertex list is obtained; A second region of interest vertex list is obtained based on the first region of interest vertex list and the group test junction coefficients; as well as Based on the multiple lens images, the third principal vertex list, and the second region of interest vertex list, the cumulative pixel value error between all measurement areas and the cross section is measured.

20. The method according to claim 14, characterized in that, The optimal cohesion coefficient for obtaining the control points of all measurement areas includes: Based on a selected control point, a selected decision group is determined; and The temporary bonding coefficient of the selected control point is determined based on the group test bonding coefficient and the local minimum of the cumulative pixel value error between each control point in the selected decision group, wherein the selected decision group includes the selected control point and one or two neighboring control points.

21. The method according to claim 20, characterized in that, The optimal cohesion coefficient for obtaining the control points of all measurement areas also includes: When the transverse segment is a single block The test bonding coefficient corresponding to the overall minimum value of the cumulative pixel value error of the cross section is defined as the temporary bonding coefficient of the cross section; and When the cross section overlaps with a first measurement area or occupies more than half of the area of ​​a second measurement area, the temporary bonding coefficient of the control point located in the first measurement area is replaced by the temporary bonding coefficient of the cross section. The second measurement area is adjacent to the first measurement area.

22. The method according to claim 20, characterized in that, The optimal cohesion coefficient for obtaining the control points of all measurement areas also includes: When the transverse segment is divided into multiple segments and the multiple segments overlap with multiple measurement areas Based on a selected segment, a selected decision group is determined; as well as The temporary bonding coefficient of the selected segment is determined based on the group test bonding coefficient and the local minimum of the cumulative pixel value error among the segments in the selected decision group, wherein the selected decision group includes the selected segment and one or two neighboring segments.

23. The method according to claim 22, characterized in that, The optimal cohesion coefficient for obtaining the control points of all measurement areas also includes: When the transverse segment is divided into multiple segments and the multiple segments overlap with multiple measurement areas After the step of determining the temporary engagement coefficient of the selected segment, when one of the plurality of segments overlaps with a first measurement area or occupies more than half of the area of ​​a second measurement area, the temporary engagement coefficient of the control point located in the first measurement area is replaced by the temporary engagement coefficient of the one of the plurality of segments. The second measurement area is adjacent to the first measurement area.

24. The method according to claim 14, characterized in that, The set of test coupling coefficients is related to the offset of the center of one lens of the multi-lens camera relative to the center of the camera system.

Images