Image processing apparatus and image processing method for generating a layout including a plurality of images

Abstract

An image processing device and an image processing method for generating a layout of multiple images are provided. The method includes: dividing the layout into multiple tiles; selecting a first tile located at an intersection of a first display region and a second display region from the multiple tiles, and cropping a first sub-block from the first tile; mapping the first sub-block to a correction map to obtain a mapped region; in response to a third vertex in the mapped region corresponding to a first vertex and a second vertex, performing an interpolation operation on a first data structure and a second data structure to obtain a third data structure corresponding to the third vertex; updating a vertex list according to the third vertex and the third data structure to generate a combined vertex list; and mapping an input image to the layout according to the combined vertex list to generate an output image.

Description

Technical Field

[0001] This invention relates to an image processing technology, and more particularly to an image processing apparatus and method for generating a layout comprising multiple images. Background Technology

[0002] Patents with publication number "US10104288" and "TW I622021" disclose a method and apparatus for generating panoramic images with stitching functionality, which can convert camera images into panoramic images using a data structure defined by a vertex list. However, the aforementioned patents can only output a single panoramic image.

[0003] Currently, many software applications (such as video conferencing software or monitoring systems) require outputting various layouts, and each layout needs to display multiple different images. For example, video conferencing software layouts may need to display multiple images including different participants. Since the aforementioned patent can only output a single panoramic image, it cannot meet the diverse layout requirements. Summary of the Invention

[0004] The present invention provides an image processing apparatus and an image processing method that can generate a layout including multiple images.

[0005] An image processing apparatus for generating a layout comprising multiple images according to the present invention includes: a transceiver for acquiring a correction map corresponding to an input image and a vertex list, wherein the vertex list includes a first vertex on the correction map, a second vertex on the correction map, a first data structure corresponding to the first vertex, and a second data structure corresponding to the second vertex; a storage medium for storing multiple modules; and a processor coupled to the storage medium and the transceiver, and accessing and executing the multiple modules, wherein the multiple modules include: a region of interest combiner for receiving layout information through the transceiver to determine a first display area and a second display area in the layout; a frame tile segmentation unit for segmenting the layout into multiple tiles; and a tile separator for separating the multiple tiles. The system selects a first tile located at the boundary between the first and second display areas, and cuts out a first sub-block corresponding to the first display area from the first tile; a coordinate converter maps the first sub-block to a calibration map to obtain a mapped area; an attribute interpolator performs interpolation on a first data structure and a second data structure in response to the correspondence between the third vertex in the mapped area and the first and second vertices to obtain a third data structure corresponding to the third vertex; a vertex generator updates the vertex list based on the third vertex and the third data structure to generate a combined vertex list; and an image processing module maps the input image to the layout based on the combined vertex list to generate an output image, and outputs the output image through a transceiver.

[0006] In one embodiment of the present invention, the above-described plurality of modules further include: a primitive assembly unit, which divides the mapping region into a plurality of primitives according to the size of the mapping region, wherein the plurality of primitives includes a first primitive having a third vertex.

[0007] In one embodiment of the present invention, the above-mentioned multiple modules further include: a primitive assembly unit, which determines whether the mapped region contains a splicing region of multiple original images based on a vertex list, and in response to the mapping region containing a splicing region, divides the mapping region into multiple primitives according to the size of the splicing region, wherein the multiple primitives include a first primitive having a third vertex.

[0008] In one embodiment of the present invention, each of the plurality of primitives is a geometric structure having vertices.

[0009] In one embodiment of the present invention, the layout information described above includes a first original display area and a second original display area in the layout, wherein the region of interest combiner responds to the partial overlap between the first original display area and the second original display area by dividing the first original display area into a plurality of sub-display areas containing the first display area, wherein the plurality of sub-display areas do not overlap with each other, and each of the plurality of sub-display areas does not overlap with the second original display area.

[0010] In one embodiment of the present invention, the layout information described above includes a first priority order corresponding to a first original display area and a second priority order corresponding to a second original display area, wherein the region of interest combiner responds to the first priority order being lower than the second priority order by segmenting the first original display area.

[0011] In one embodiment of the present invention, the region of interest combiner determines the projection method based on the layout information, wherein the coordinate converter maps the first sub-block to the calibration map according to the projection method to obtain the mapped region.

[0012] In one embodiment of the present invention, the attribute interpolator described above determines a first weight corresponding to the first vertex and a second weight corresponding to the second vertex based on the distance between the third vertex and the first vertex in a first direction.

[0013] In one embodiment of the present invention, the attribute interpolator described above calculates a first product of a first attribute value and a first weight in a first data structure and a second product of a second attribute value and a second weight in a second data structure to generate a third data structure.

[0014] In one embodiment of the present invention, the first data structure includes a first scaling factor and a first blending weight corresponding to a first vertex, and the second data structure includes a second scaling factor and a second blending weight corresponding to a second vertex, wherein the attribute interpolator calculates a first product of the first blending weight and the first weight to obtain or update a first accumulated value corresponding to the first scaling factor, and calculates a second product of the second blending weight and the second weight to obtain or update a second accumulated value corresponding to the second scaling factor, wherein the attribute interpolator selects a first scaling factor from the first scaling factor and the second scaling factor in response to the first accumulated value being greater than the second accumulated value to generate a third data structure.

[0015] In one embodiment of the present invention, the storage medium further stores a first storage array and a second storage array. After obtaining the first product, the attribute interpolator determines whether the first scaling factor matches a first address in the first storage array and whether the first address stores data. If the first address already stores data, in response to the determination that the first scaling factor matches the first address, the attribute interpolator adds the first product to the second address in the second storage array corresponding to the first address. If the first address already stores data, in response to the determination that the first scaling factor does not match the first address, the attribute interpolator stores the first scaling factor at a third address in the first storage array and adds the first product to the fourth address in the second storage array corresponding to the third address. If the first address does not store data, the attribute interpolator stores the first scaling factor in the first address and adds the first product to the second address.

[0016] In one embodiment of the present invention, the storage medium further stores a first storage array and a second storage array, wherein a first scaling factor matches a first address in the first storage array, and an attribute interpolator adds a first multiplication accumulation to a second position in the second storage array corresponding to the first address according to a lookup table.

[0017] In one embodiment of the invention, the vertex generator generates a current frame header corresponding to the current frame of the output image based on the combined vertex list, wherein the current frame header indicates a third display area in the layout, wherein the attribute interpolator responds to the fourth display area in the next frame of the output image matching the third display area by copying the data in the current frame header to generate a new combined vertex list corresponding to the next frame, wherein the data includes a fourth vertex corresponding to the third display area and a fourth data structure corresponding to the fourth vertex.

[0018] In one embodiment of the present invention, the input image described above is an equidistant rectangular projection image.

[0019] In one embodiment of the present invention, the above-mentioned layout information includes at least: the coordinates, width and height of a first region of interest corresponding to the source image and the coordinates, width and height of a display area corresponding to the target image.

[0020] An image processing method for generating a layout comprising multiple images according to the present invention includes: obtaining a calibration map corresponding to an input image and a vertex list, wherein the vertex list includes a first vertex on the calibration map, a second vertex on the calibration map, a first data structure corresponding to the first vertex, and a second data structure corresponding to the second vertex; receiving layout information to determine a first display area and a second display area in the layout; dividing the layout into multiple tiles; selecting a first tile located at the boundary between the first display area and the second display area from the multiple tiles, and cutting out a first sub-block corresponding to the first display area from the first tile; mapping the first sub-block to the calibration map to obtain a mapped area; performing an interpolation operation on the first data structure and the second data structure in response to a third vertex in the mapped area corresponding to the first vertex and the second vertex to obtain a third data structure corresponding to the third vertex; updating the vertex list according to the third vertex and the third data structure to generate a combined vertex list; and mapping the input image to the layout according to the combined vertex list to generate an output image, and outputting the output image.

[0021] The present invention provides an image processing apparatus for generating a layout comprising multiple images, comprising a processor. The processor is configured to perform: acquiring a correction map corresponding to an input image and a vertex list, wherein the vertex list includes a first vertex on the correction map, a second vertex on the correction map, a first data structure corresponding to the first vertex, and a second data structure corresponding to the second vertex; receiving layout information to determine a first display area and a second display area in the layout; dividing the layout into multiple tiles; selecting a first tile located at the boundary between the first and second display areas from the multiple tiles, and cutting a first sub-block corresponding to the first display area from the first tile; mapping the first sub-block to the correction map to obtain a mapped area; performing an interpolation operation on the first and second data structures in response to a third vertex in the mapped area corresponding to the first and second vertices to obtain a third data structure corresponding to the third vertex; updating the vertex list according to the third vertex and the third data structure to generate a combined vertex list; and mapping the input image to the layout according to the combined vertex list to generate an output image, and outputting the output image.

[0022] Based on the above, the image processing apparatus of the present invention can divide a layout into multiple tiles and map the input image to the display area of ​​the layout using multiple vertices forming tiles as units. When the display area includes boundaries or splicing areas, the image processing apparatus further cuts the tiles into smaller sub-blocks or primitives, thereby increasing the number of vertices in the boundaries or splicing areas. Thus, when the image processing apparatus converts the input image into an output image based on the method disclosed in patent publication number "US10104288" or patent publication number "TWI622021" according to the vertex data structure, the image processing apparatus can render the boundaries or splicing areas in the output image based on more vertex information. Therefore, the image processing apparatus can improve the distortion problem of boundaries or splicing areas while maintaining the efficiency of generating the output image. Attached Figure Description

[0023] Figure 1 A schematic diagram of an image processing apparatus is shown according to an embodiment of the present invention.

[0024] Figure 2A A schematic diagram of the output image corresponding to the layout is shown according to an embodiment of the present invention.

[0025] Figure 2B A schematic diagram illustrating a layout divided into multiple tiles is shown according to an embodiment of the present invention.

[0026] Figure 2C A schematic diagram illustrating the cutting of multiple sub-blocks from a self-drawing brick according to an embodiment of the present invention is shown.

[0027] Figure 3A A schematic diagram of multiple modules in a storage medium is shown according to an embodiment of the present invention.

[0028] Figure 3B A schematic diagram of the pipeline of an instant combiner is shown according to an embodiment of the present invention.

[0029] Figure 4 A schematic diagram of a calibration diagram is shown according to an embodiment of the present invention.

[0030] Figure 5 A schematic diagram illustrating the overlap pattern of the original display area 510 and the original display area 520 according to an embodiment of the present invention is shown.

[0031] Figure 6 A schematic diagram of perspective projection is shown according to an embodiment of the present invention.

[0032] Figure 7A A schematic diagram of a mapping region corresponding to a sub-block is shown according to an embodiment of the present invention.

[0033] Figure 7B and7C A schematic diagram illustrating the division of a tile into multiple elements is shown according to an embodiment of the present invention.

[0034] Figure 8A , 8B The diagram above illustrates a schematic diagram of brick 410 according to an embodiment of the present invention.

[0035] Figure 9 A flowchart illustrating an image processing method for generating a layout comprising multiple images is shown according to an embodiment of the present invention.

[0036] Explanation of reference numerals in the attached figures

[0037] 10: Image processing device;

[0038] 110: Processor;

[0039] 120: Storage medium;

[0040] 121: Correspondence generator;

[0041] 122: Calibration diagram generator;

[0042] 123: Region of Interest Combiner;

[0043] 124: Instant combiner;

[0044] 125: Image capturing module;

[0045] 126: Image receiving module;

[0046] 127: Image processing module;

[0047] 130: Transceiver;

[0048] 200: Layout;

[0049] 21: Boundary point;

[0050] 210, 220, 410, 61: Drawing bricks;

[0051] 211, 212, 213: Sub-blocks;

[0052] 2111, 6111: Vertices;

[0053] 22: Splicing line;

[0054] 31: Frame brick segmentation unit;

[0055] 32: Brick separator;

[0056] 33: Coordinate converter;

[0057] 34: Primitive Assembly Unit;

[0058] 35: Calibration chart finder;

[0059] 36: Attribute interpolator;

[0060] 37: Vertex generator;

[0061] 400: Calibration diagram;

[0062] 510, 520: Original display area;

[0063] 521, 522: Sub-display areas;

[0064] 530: Overlapping mode;

[0065] 60: Equator;

[0066] 611, 620: Mapping areas;

[0067] 62, 71, 72, 73: Graphical elements;

[0068] 621: Splicing area;

[0069] S901, S902, S903, S904, S905, S906, S907, S908: Steps. Detailed Implementation

[0070] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same element symbols are used in the drawings and description to denote the same or similar parts.

[0071] Figure 1 A schematic diagram of an image processing apparatus 10 is illustrated according to an embodiment of the present invention. The image processing apparatus 10 may include a processor 110, a storage medium 120, and a transceiver 130.

[0072] Processor 110 may be, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microcontroller (MCU), microprocessor, digital signal processor (DSP), programmable controller, application-specific integrated circuit (ASIC), graphics processing unit (GPU), image signal processor (ISP), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field-programmable gate array (FPGA), or other similar elements or combinations thereof. Processor 110 may be coupled to storage medium 120 and transceiver 130, and access and execute multiple modules and various applications stored in storage medium 120.

[0073] Storage medium 120 may be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD), or similar element or combination thereof, for storing multiple modules or various applications that can be executed by processor 110.

[0074] Transceiver 130 transmits and receives signals wirelessly or via a wired connection. Transceiver 130 can also perform operations such as low-noise amplification, impedance matching, mixing, up or down frequency conversion, filtering, amplification, and similar functions.

[0075] The image processing device 10 can generate an output image corresponding to the layout according to the user's layout requirements, such as... Figure 2A As shown. In Figure 2AIn this layout, the output image layout 200 can include multiple display areas, such as display area A, display area B, and display area C, and different display areas can display regions of interest (ROIs) from the same or different input images. For example, assuming layout 200 is the layout of remote conferencing software, display areas A, B, and C can respectively display images of different participants, such as user A, user B, and user C. The images of user A and user C can correspond to different ROIs of the same input image. The image of user B can correspond to ROIs of different input images.

[0076] Figure 3A A schematic diagram of multiple modules in a storage medium 120 is illustrated according to an embodiment of the present invention. The processor 110 can access and execute the multiple modules in the storage medium 120 to perform various functions of the image processing apparatus 10. In this embodiment, the storage medium 120 may include multiple modules such as a correspondence generator 121, a calibration map builder 122, a region of interest (ROI) composer 123, a real-time composer 124, an image capturing module 125, an image receiving module 126, and an image processing module 127, the functions of which will be described later.

[0077] The correspondence generator 121 generates an original vertex list corresponding to the input image. The method for generating the original vertex list can be found in patent publications US10104288 and TWI622021. In one embodiment, the correspondence generator 121 receives the original vertex list via transceiver 130. The image capturing module 125 or the image receiving module 126 acquires the original image via transceiver 130 and generates the input image. Specifically, the image capturing module 125 can be communicatively connected to one or more image capturing devices via transceiver 130 to receive original images that can be used by one or more image capturing devices to generate equirectangular projection images. On the other hand, the image receiving module 126 can be communicatively connected to one or more electronic devices via transceiver 130 to receive original images of other types not used for generating equirectangular projection images. The input image may include an isometric rectangular projection image converted from the original image obtained by the image capturing module 125, or other types of images converted from the original image obtained by the image receiving module 126. In one embodiment, the correspondence generator 121 may generate a corresponding original vertex list based on the input image from the image capturing module 125 or the image receiving module 126, as described in patents US10104288 and TW1622021. In one embodiment, a single input image may include one or more textures, where the texture is, for example, a camera image (i.e., the original image). For example, the isometric rectangular projection image as the input image may be stitched together from multiple camera images captured by multiple cameras (or an isometric rectangular projection image obtained by converting camera images), so the input image may include multiple textures.

[0078] The original vertex list records multiple vertices and multiple data structures corresponding to each vertex. After obtaining the original vertex list, the calibration map generator 122 can generate a calibration map corresponding to the input image based on the original vertex list. For example, if the resolution of the input image is 1920x1080, the correspondence generator 121 can generate a calibration map corresponding to the resolution of 1920x1080. The original vertex list may contain the mapping relationship between the input image and the layout 200 (or the output image) and can be used to map a specific input image to a specific display area (or a specific sub-display area) in the layout 200.

[0079] The calibration graph may contain multiple vertices recorded in the original vertex list. Figure 4A schematic diagram of a correction graph 400 is illustrated according to an embodiment of the present invention. The correction graph 400 may include 144 vertices and may include 144 data structures corresponding to each of the 144 vertices. Multiple vertices may form a tile. For example, the correction graph 400 may include vertices v0, v1, v2, and v3, and may include multiple data structures corresponding to each of vertices v0, v1, v2, and v3. Vertices v0, v1, v2, and v3 may form a tile 410.

[0080] Table 1 is an example of the information contained in the original vertex list, where the original vertex list (or the correction graph 400) is assumed to contain n vertices. The memory address of the data structure in Table 1 can point to the data structure of a specific vertex. Table 2 is an example of the data structure of the vertices in the correction graph 400, where each attribute in Table 1 and Table 2 can be obtained from the original vertex list. Patent publication number "US10104288" or patent publication number "TW I622021" describes specific implementations of rendering an input image to generate an output image based on the vertex data structure.

[0081] Table 1

[0082]

[0083] Table 2

[0084]

[0085] On the other hand, refer to Figure 3A The region of interest combiner 123 can receive layout information edited by the user via transceiver 130, wherein the layout information includes the mapping relationship between the source image (i.e., the input image) and the target image (i.e., the output image). Specifically, the region of interest combiner 123 can obtain the ROI window descriptor from the layout information. Table 3 shows an example of an ROI window descriptor. The region of interest combiner 123 can obtain relevant information about one or more display areas from the ROI window descriptor, and determine, based on the information, how to... Figure 2AThe layout 200 is shown. For example, the position and size of the display area (e.g., display area A, B, or C) can be determined based on the attributes "(dst_x, dst_y)" and "(dst_w, dst_h)". It is worth noting that although the examples in Table 3 only record a set of mapping relationships between ROI windows in the source image and display areas in the target image, the ROI window descriptor can also contain multiple sets of mapping relationships. In other words, the user can define multiple ROI windows on one or more input images, or multiple display areas on the output image (or layout 200). The projection methods in Table 3 may include, but are not limited to, rendering methods such as perspective projection, linear crop projection, or homograph transform. Linear crop projection can be used to crop and scale isometric rectangular projection images to produce output images with a larger viewing angle.

[0086] Table 3

[0087]

[0088] In one embodiment, the region of interest combiner 123 can check whether the attributes "(src_x, src_y)" and "(src_w, src_h)" conform to the specifications based on the attribute "projection". For example, if the attribute "projection" indicates that the projection method is perspective projection, the region of interest combiner 123 can check the attributes "(src_x, src_y)" and "(src_w, src_h)" to ensure that the view angle of the ROI window described by the two attributes is less than 180 degrees. If the view angle of the ROI window is greater than or equal to 180 degrees, the region of interest combiner 123 can determine that the layout information is incorrect and can output a warning message through the transceiver 130.

[0089] In one embodiment, the region of interest combiner 123 can check whether the attributes "(dst_x, dst_y)" and "(dst_w, dst_h)" conform to the specifications based on the range of the output image (or layout 200). For example, the region of interest combiner 123 can check the attributes "(dst_x, dst_y)" and "(dst_w, dst_h)" to ensure that the display area (or sub-display area) described by the two attributes does not exceed the default range of the output image. If the display area (or sub-display area) exceeds the default range of the output image, the region of interest combiner 123 can determine that the layout information is incorrect and can output a warning message through the transceiver 130.

[0090] In one embodiment, the layout information may include multiple original display areas in layout 200. If multiple original display areas overlap in layout 200, the region of interest combiner 123 divides a specific original display area according to the priority order of the multiple overlapping original display areas, thereby generating multiple display areas in layout 200. Figure 5 A schematic diagram illustrating the overlap pattern of the original display area 510 and the original display area 520 according to an embodiment of the present invention is shown. Assuming that the layout information includes the original display area 510 and the original display area 520, and that the layout information records that the priority of the original display area 520 is lower than that of the original display area 510, then the region of interest combiner 123 can select from the original display area 510 and the original display area 520 to divide the original display area 520.

[0091] Specifically, the Region of Interest (ROI) combiner 123 can segment one or more sub-display areas from the original display area 520 in response to partial overlap between the original display area 510 and the original display area 520, wherein the original display areas do not overlap with each other, and each of the original display areas does not overlap with the original display area 510. Taking overlap mode 530 as an example, if the original display area 520 partially overlaps with the original display area 510, and the priority of the original display area 520 is lower than that of the original display area 510, then the ROI combiner 123 can segment sub-display areas 521 and 522 from the original display area 520. Sub-display areas 521 and 522 do not overlap with each other, and neither sub-display area 521 nor sub-display area 522 overlaps with the display area 510. In one embodiment, the ROI combiner 123 can determine the priority of the display areas based on the identifier of the ROI window. For example, if the order of the identifier of the ROI window corresponding to the sub-display area 521 (e.g., ID#2) is greater than the order of the identifier of the ROI window corresponding to the sub-display area 522 (e.g., ID#1), then the priority of the sub-display area 521 can be higher than the priority of the sub-display area 522.

[0092] The region of interest combiner 123 can use sub-display region descriptors to represent one or more generated sub-display regions. Table 4 shows examples of sub-display region descriptors. When the region of interest combiner 123 generates multiple sub-display regions, it can generate multiple sub-display region descriptors corresponding to each of the multiple sub-display regions. Figure 5Taking the overlap mode 530 as an example, the region of interest combiner 123 can generate corresponding sub-display region descriptors for sub-display regions 521 and 522 respectively. Since sub-display regions 521 and 522 correspond to the same original display region 520 (or the same ROI window), the attribute "id1" of the descriptor of sub-display region 521 is the same as the attribute "id1" of the descriptor of sub-display region 522. The attributes "id1" and "id2" of the current frame can be recorded in the cache in the storage medium 120 as the attributes "cache_ref_id1" and "cache_ref_id2" of the next frame. In other words, the current frame will become the reference frame for the next frame.

[0093] Table 4

[0094]

[0095] After generating one or more sub-display areas, the Region of Interest (ROI) combiner 123 can update the ROI window descriptors shown in Table 3 to the ROI window descriptors shown in Table 5 based on the one or more sub-display areas. The ROI combiner 123 can then transmit the updated ROI window descriptors to the instant combiner 124.

[0096] Table 5

[0097]

[0098] Figure 3B A schematic diagram of the pipeline of the instant combiner 124 is shown according to an embodiment of the present invention. The pipeline of the instant combiner 124 may include a frame tile partition unit 31, a tile splitter 32, a coordinate transformer 33, a primitive assembly unit 34, a calibration map searcher 35, an attribute interpolator 36, and a vertex generator 37. The three inputs of the pipeline include a ROI window descriptor input to the frame tile partition unit 31, a raw vertex list input to the calibration map searcher 35, and a calibration map input to the calibration map searcher 35. The pipeline can generate a composed vertex list based on the above three inputs. The vertex generator 38 can output the composed vertex list to the image processing module 127.

[0099] Please also refer to Figures 2A-2CThe frame brick segmentation unit 31 can generate, based on the ROI window descriptor, such as... Figure 2A The layout 200 shown can be divided into multiple tiles. Figure 2B A schematic diagram illustrating the division of layout 200 into multiple tiles is shown according to an embodiment of the present invention.

[0100] The tile separator 32 can select the tile 210 located at the boundary 21 between display area A and display area B from a plurality of tiles. Then, the tile separator 32 can cut multiple sub-blocks from the tile 210, wherein the multiple sub-blocks may include a sub-block 211 corresponding to display area A, a sub-block 212 corresponding to the boundary 21, and a sub-block 213 corresponding to display area B, such as... Figure 2C As shown, vertex 2111 is one of the vertices of sub-block 211.

[0101] Coordinate converter 33 can map each point in the tiles or sub-blocks of the display area to the corresponding calibration map to obtain the mapped area on the calibration map. Coordinate converter 33 can also map the vertices of the tiles or sub-blocks of the display area to the corresponding calibration map to obtain the mapped vertices on the calibration map, wherein the mapped vertices can constitute the mapped area.

[0102] The following example illustrates mapping a tile or sub-block in display area A to a calibration map. Assuming display area A corresponds to calibration map 400, coordinate converter 33 can map the tile or sub-block (e.g., sub-block 211) in display area A to calibration map 400, thereby generating a mapped area in calibration map 400 corresponding to display area A. Specifically, region of interest combiner 123 can determine the projection method as described in Table 5 based on layout information. Coordinate converter 33 can map multiple vertices of a tile or sub-block in display area A to calibration map 400 according to the projection method to generate multiple mapped vertices corresponding to multiple vertices, wherein the multiple mapped vertices can constitute a mapped area.

[0103] Taking sub-block 211 in display area A as an example, if the projection method is linear clipping projection, then coordinate converter 33 can map the vertices of sub-block 211 to the calibration map 400 according to equations (1) and (2), where i represents the index of the ROI window (or the identifier of the ROI window), x' represents the abscissa corresponding to the calibration map 400 (i.e., the abscissa of the mapped area of ​​sub-block 211), x represents the abscissa corresponding to the output image (i.e., the abscissa of sub-block 211), src_x represents the abscissa of the origin of the ROI window in the source image (i.e., the input image), and src_w represents the abscissa of the origin of the ROI window in the source image. The coordinates are defined as follows: dst_x represents the x-coordinate of the origin of the display area in the target image (i.e., the output image); dst_w represents the width of the display area in the target image; y' represents the y-coordinate corresponding to the calibration graph 400 (i.e., the y-coordinate of the mapped area of ​​sub-block 211); y represents the y-coordinate corresponding to the output image (i.e., the y-coordinate of sub-block 211); src_y represents the y-coordinate of the origin of the ROI window in the source image; src_h represents the height of the ROI window in the source image; dst_y represents the y-coordinate of the origin of the display area in the target image; and dst_h represents the height of the display area in the target image. In the above coordinates, the origin of the coordinate system of the output image can be the center point of the display area A, and the origin of the coordinate system of the calibration graph 400 can be the center point of the mapped area of ​​the display area A.

[0104]

[0105]

[0106] Figure 6A schematic diagram of perspective projection is illustrated according to an embodiment of the present invention. If the projection method is perspective projection, the coordinate converter 33 can map the display area A to the correction diagram 400 to obtain the mapped area A' according to equations (3) to (8), wherein the center point of the mapped area A' corresponds to the center point of the display area A. f represents the focal length used for perspective projection of the display area A, H represents half of the horizontal width of the display area A, θ represents the horizontal viewing angle or half of the horizontal width of the mapped area A', δ represents the distance between the center point of the mapped area A' and the equator 60 of the correction diagram 400, x represents the horizontal coordinate corresponding to the display area A, y represents the vertical coordinate corresponding to the display area A, α represents the horizontal distance between the mapping point and the center point of the mapped area A' (the horizontal distance can be converted into the horizontal angle of the equidistant rectangular projection image), and β represents the vertical distance between the mapping point and the equator 60 (the vertical distance can be converted into the vertical angle of the equidistant rectangular projection image). The information for attributes dst_w, src_w, src_y, and src_h can be derived from the ROI window descriptors shown in Table 3. EQT_VFOV can be the maximum vertical angle of the equidistant rectangular projected image, EQT_HFOV can be the maximum horizontal angle of the equidistant rectangular projected image, EQT_Width can be the width of the equidistant rectangular projected image, and EQT_Height can be the height of the equidistant rectangular projected image. Parameters such as EQT_VFOV, EQT_HFOV, EQT_Width, or EQT_Height can be associated with the hardware specifications of the image capturing device.

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] After mapping the vertices of the tiles or sub-blocks of the display area to the correction map 400 to generate mapped vertices and the mapped area, the primitive assembly unit 34 can divide the mapped area into multiple tiles according to the size of the mapped area, and each tile does not overlap with each other. Taking sub-block 211 as an example, Figure 7AA schematic diagram of a mapping region 611 corresponding to a sub-block 211 is illustrated according to an embodiment of the present invention, wherein the mapping region 611 is composed of a plurality of mapped vertices respectively corresponding to a plurality of vertices of the sub-block 211. For example, the vertices 6111 of the mapping region 611 are generated by mapping the vertices 2111 of the sub-block 211 to the correction diagram 400. The primitive assembly unit 34 can divide the mapping region 611 into a plurality of tiles 61 according to the size of the mapping region 611, and each tile 61 does not overlap with each other.

[0114] In one embodiment, the primitive assembly unit 34 can determine whether the mapped region contains spliced ​​regions of multiple original images based on information recorded in the calibration map or the original vertex list. If the vertex data structure (i.e., the attribute "ntex" as shown in Table 2) records that the vertex corresponds to multiple textures, the primitive assembly unit 34 can determine whether the mapped region composed of the vertices contains spliced ​​regions. If the mapped region contains spliced ​​regions, the primitive assembly unit 34 can further divide the tiles in the calibration map into multiple primitives. Specifically, the primitive assembly unit 34 can determine the shape of the primitives based on the size of the spliced ​​regions, thereby dividing the mapped region into multiple primitives.

[0115] Figure 7B and 7C A schematic diagram illustrating the division of a tile into multiple elements is shown according to an embodiment of the present invention. (Refer to...) Figure 2B and 7B Assume that coordinate converter 33 maps the vertices of tile 220 in layout 200 to a correction map to generate a mapped region 620, and assume that tile 220 (or display area C) corresponds to correction map 400. Since tile 220 contains the stitching line 22 of two original images, mapped region 620 contains a stitching region 621 corresponding to the stitching line 22. Primitive assembly unit 34 can determine that mapped region 620 contains stitching region 621 based on the original vertex list corresponding to correction map 400. If at least one vertex associated with a specific point in correction map 400 corresponds to multiple textures (i.e., multiple textures corresponding to stitching region 621), primitive assembly unit 34 can determine that the specific point is located in stitching region 621. Otherwise, primitive assembly unit 34 can determine that the specific point is located outside stitching region 621.

[0116] Specifically, if a specific point on the calibration graph 400 is located on an existing vertex of the calibration graph 400 (i.e., a vertex recorded in the original vertex list), the primitive assembly unit 34 can determine that the existing vertex is associated with the specific point, and determine whether the specific point is located in the splicing region 621 based on the existing vertex. If the specific point is located on the line connecting two existing vertices of the calibration graph 400, the primitive assembly unit 34 can determine whether one of the two existing vertices corresponds to multiple textures. If one of the two existing vertices corresponds to multiple textures, the primitive assembly unit 34 can determine that the specific point is located in the splicing region 621. If the specific point is located within a tile (e.g., tile 410) of the calibration graph 400, the primitive assembly unit 34 can determine whether the four existing vertices on the calibration graph 400 closest to the specific point (i.e., the four vertices recorded in the original vertex list and surrounding the specific point) correspond to multiple textures. If one of the four existing vertices corresponds to multiple textures, the primitive assembly unit 34 can determine that the specific point is located in the splicing region 621.

[0117] After determining that the mapping region 620 includes the splicing region 621, the primitive assembly unit 34 can divide the mapping region 620 (or the splicing region 621) into multiple primitives according to the size of the splicing region 621, wherein the multiple primitives do not overlap. The resulting primitives can be pieced together to form the complete splicing region 621. Figure 7B For example, assuming the splicing area 621 is rectangular, the primitive assembly unit 34 can divide the tile 61 that partially overlaps with the splicing area 621 into two rectangular primitives 62. The splicing area 621 can be assembled from three tiles 61 and three primitives 62. Figure 7C For example, assuming the splicing area 621 is a parallelogram, the primitive assembly unit 34 can divide the mapping area 620 into multiple triangular primitives, such as primitive 71, primitive 72, and primitive 73. The splicing area 621 in the mapping area 620 can be assembled from 12 triangular primitives.

[0118] After the primitive assembly unit 34 divides the mapping region 620 into multiple primitives, the correction graph 400 may contain several types of vertices, including existing vertices (i.e., vertices recorded in the original vertex list), mapped vertices in the mapping region (e.g., mapping region 620) (e.g., vertices generated by mapping the vertices of tile 22 in layout 200 to the correction graph 400), and primitive vertices (e.g., vertices of primitive 62). Since the original vertex list only records the data structure of existing vertices, the data structure of mapped vertices or primitive vertices is unknown. Accordingly, the attribute interpolator 36 can perform interpolation operations based on the data structure of existing vertices to obtain the data structure of mapped vertices or primitive vertices.

[0119] Specifically, the correction graph finder 35 can determine which existing vertex in the correction graph 400 is associated with a specific vertex (i.e., a mapped vertex or a primitive vertex) in the mapping region. For example... Figure 4 Taking the illustrated tile 410 as an example, the correction graph finder 35 can determine which vertex of the tile 410 is associated with the specific vertex based on the position of the specific vertex in the tile 410. Figure 8A , 8B Figure 8C illustrates a schematic diagram of a tile 410 according to an embodiment of the present invention, wherein vertices v0, v1, v2, and v3 of tile 410 are existing vertices recorded in the original vertex list. (See also...) Figure 8A If vertex v' in the mapped region (e.g., mapped region 620) lies on vertex v1 of the correction graph 400, then the correction graph finder 35 can determine that vertex v1 is associated with vertex v'. (See reference...) Figure 8B If vertex v' lies on the line connecting two existing vertices (i.e., vertices v0 and v1) in the correction graph 400, then the correction graph finder 35 can determine that vertices v0 and v1 are associated with vertex v'. (Refer to...) Figure 8C If vertex v' is located within tile 410 of the correction graph 400, the correction graph finder 35 can determine that the four existing vertices (i.e., vertices v0, v1, v2 and v') closest to vertex v' on the correction graph 400 are associated with vertex v'.

[0120] If vertex v' is associated with a single existing vertex, then attribute interpolator 36 can produce a data structure for vertex v', where the data structure of vertex v' is the same as that of the existing vertex. Figure 8A For example, attribute interpolator 36 can set the data structure of v' to the data structure of v1. On the other hand, if vertex v' is associated with multiple existing vertices, attribute interpolator 36 can perform interpolation operations on multiple data structures corresponding to the multiple existing vertices to obtain the data structure corresponding to vertex v'.

[0121] by Figure 8CFor example, attribute interpolator 36 can perform interpolation operations on multiple data structures corresponding to vertices v0, v1, v2 and v3 respectively to obtain the data structure of vertex v'. In one embodiment, the attribute interpolator 36 can perform a bilinear interpolation operation as shown in equation (9) to obtain the attribute "(u, v)" or attribute "alpha" in the data structure of vertex v', where dx is the distance in the x direction between the origin of the tile 410 (i.e., the vertex v0 in the upper left corner) and vertex v', dy is the distance in the y direction between the origin of the tile 410 and vertex v', v'.k is the attribute value of vertex v' (e.g., the attribute value of attribute (u, v) or alpha), v0.k is the attribute value of vertex v0, B[0] is the weight corresponding to vertex v0, v1.k is the attribute value of vertex v1, B[1] is the weight corresponding to vertex v1, v2.k is the attribute value of vertex v2, B[2] is the weight corresponding to vertex v2, v3.k is the attribute value of vertex v3, and B[3] is the weight corresponding to vertex v3.

[0122]

[0123] In the data structure of vertex v', not all attribute values ​​are obtained by performing interpolation. In one embodiment, attribute interpolator 36 can calculate the cumulative value of scaling factors for each overlapping control region (corresponding to attribute sca_bland[i]) in the data structure. Attribute interpolator 36 can calculate the cumulative value w[i] of scaling factors according to equation (10), where i represents the index of the overlapping control region (i=1~m), j represents the index of the existing vertex associated with vertex v' (j=0~3), w[i] represents the cumulative value of scaling factors corresponding to the i-th overlapping control region, v[j].sca_blend[i] represents the blend weight indicated by the attribute "sca_blandi (i=1~m)" of vertex vj, and B[j] represents the weight corresponding to vertex vj.

[0124]

[0125] The attribute interpolator 36 can select one or more scaling factors with larger cumulative values ​​from a plurality of scaling factors to assign to vertex v', thereby generating a data structure for vertex v'. If the plurality of scaling factors includes a first scaling factor and a second scaling factor, the attribute interpolator 36 can preferentially select the first scaling factor from the first and second scaling factors to generate the data structure for vertex v' in response to the first cumulative value corresponding to the first scaling factor being greater than the second cumulative value corresponding to the second scaling factor. The larger the cumulative value of the scaling factor, the greater the influence of the scaling factor on vertex v'. The smaller the cumulative value of the scaling factor, the smaller the influence of the scaling factor on vertex v'. The attribute interpolator 36 can set the attribute "sca_id1" of vertex v' to point to the scaling factor with the largest cumulative value, and can set the attribute "sca_id2" of vertex v' to point to the scaling factor with the second largest cumulative value. And so on, the attribute interpolator 36 can set the attribute "sca_idm" of vertex v' to point to the scaling factor with the smallest cumulative value.

[0126] For example, after calculating m accumulated values ​​w[i] (i=1~m) for m overlapping control regions, attribute interpolator 36 can select the larger n accumulated values ​​(n≤m) from the m accumulated values ​​w[i]. Taking n=2 as an example, attribute interpolator 36 can select a scaling factor corresponding to the accumulated value w[p1] in response to the accumulated value w[p1] being the largest accumulated value among the m accumulated values, and set the attributes "sca_id1" and "sca_blend1" of vertex v' according to the selected scaling factor. Attribute interpolator 36 can select a scaling factor corresponding to the accumulated value w[p2] in response to the accumulated value w[p2] being the second largest accumulated value among the m accumulated values, and set the attributes "sca_id2" and "sca_blend2" of vertex v' according to the selected scaling factor. The attribute interpolator 36 can set the attribute "sca_id1" of vertex v' as an identifier pointing to the selected scaling factor corresponding to the accumulated value w[p1], and can calculate the attribute "sca_blend1" according to equation (11). The attribute interpolator 36 can set the attribute "sca_id2" of vertex v' as an identifier pointing to the selected scaling factor corresponding to the accumulated value w[p2], and can calculate the attribute "sca_blend2" according to equation (11). k represents the index of the attribute "sca_blend" (when k=1, sca_blend[k] represents the attribute sca_blend1), j represents the index of the n selected cumulative values ​​(e.g., j=1 indicates the cumulative value w[p1]; j=2 indicates the cumulative value w[p1]), and l represents the index of the cumulative value of the selected scaling factor (e.g., when l=p1, w[l] represents the cumulative value w[p1]; when l=p2, w[l] represents the cumulative value w[p2]).

[0127]

[0128] In one embodiment, storage medium 120 includes storage array hid and storage array hist, wherein storage array hid is used to store the attribute "sca_id" of m overlapping control regions, and storage array hist is used to store the attribute "sca_blend" of m overlapping control regions.

[0129] In one embodiment, after calculating the product ( After that, the attribute interpolator 36 can determine the product. The scaling factor (corresponding to the attribute sca_id[α]) is checked against address #1 in the storage array hid, and it is determined whether address #1 in the storage array hid stores any data. If address #1 does not store any data (as shown in Table 6), the attribute interpolator 36 can store the attribute sca_id[α] indicating the scaling factor at address #1 in the storage array hid, and multiply the product... The accumulated value A is added to address #A of the storage array hist, as shown in Table 7, where address #A corresponds to address #1.

[0130] Table 6

[0131]

[0132] Table 7

[0133]

[0134] If address #1 in the storage array HID already stores data, then attribute interpolator 36 can further determine the corresponding product. The scaling factor (corresponding to the attribute sca_id[α]) matches the address #1 in the storage array hid. If it corresponds to the product... If the scaling factor matches address #1 in the storage array hid (as shown in Table 8), then the attribute interpolator 36 can multiply the product. The accumulated value A is added to address #A of the storage array hist, as shown in Table 7, where address #A corresponds to address #1.

[0135] Table 8

[0136]

[0137] If address #1 in the storage array HID already stores data, and corresponds to the product If the scaling factor does not match address #1 in the storage array hid (as shown in Table 9), then the attribute interpolator 36 can store the attribute sca_id[α] indicating the scaling factor at address #2 in the storage array hid, where no data has yet been stored, and multiply the product... The accumulated value B is added to address #B of the storage array hist, as shown in Table 10, where address #B corresponds to address #2.

[0138] Table 9

[0139]

[0140] Table 10

[0141]

[0142] In one embodiment, the storage medium 120 may store a lookup table, wherein the lookup table may contain a mapping relationship between the attribute "sca_id" and the address of the storage array hid. After calculating the product corresponding to "attribute sca_id[α]"... Then, attribute interpolator 36 can determine the address (e.g., address #1) in the storage array HID corresponding to "attribute sca_id[α]" based on the lookup table. Attribute interpolator 36 can further multiply... The product is accumulated to the corresponding address in the memory array's `hist` (e.g., address #A corresponding to address #1). Using a lookup table to calculate the accumulated value avoids the step of "checking if the product matches the address in the memory array's `hist`," thus significantly reducing computational resource consumption.

[0143] After obtaining one or more attributes "sca_id" and attribute "sca_blend" of vertex v', attribute interpolator 36 can calculate the attribute "scaling coefficient" of vertex v' according to equation (12), where i represents the index of the overlapping control region, j represents the index of the existing vertex associated with vertex v', v'.S represents the attribute value of the attribute "scaling coefficient" of vertex v', v[j].S represents the warping value corresponding to the existing vertex, and B[j] represents the weight corresponding to the existing vertex. The scaling factor represents the existing vertices. Represents the mixed weights of existing vertices, and C( ) represents the scaling factor The warping coefficient.

[0144]

[0145] In one embodiment, the processor 110 may have multiple cores. The attribute interpolator 36 may utilize each core to generate multiple data structures corresponding to multiple vertices. In this way, the image processing apparatus 10 can quickly obtain the data structure of each vertex (e.g., vertex v').

[0146] After obtaining the data structure of vertex v', vertex generator 37 can add the data structure of vertex v' to the original vertex list to update the original vertex list, thereby generating a composed vertex list. Vertex generator 37 can output the composed vertex list to image processing module 127. Image processing module 127 can map the input image to layout 200 according to the composed vertex list to generate an output image. In other words, image processing module 127 can render the input image according to the composed vertex list to generate an output image. Image processing module 127 can output the generated output image through transceiver 130. Patents with publication number "US10104288" or "TW I622021" describe specific implementation methods for rendering input images according to vertex lists to generate output images.

[0147] In one embodiment, processor 110 may obtain information from the original vertex list, ROI window descriptor, and sub-display region descriptor, and store the obtained information in the intra-frame cache of storage medium 120. Table 11 shows an example of the data structure of the intra-frame cache, wherein the data structure includes a group of attributes "ntex", "win_id", and "vtx_addr". Before attribute interpolator 36 generates a data structure for a vertex, attribute interpolator 36 may determine whether the attribute value of the vertex matches the attribute "ntex" and attribute "win_id" in any group. If the attribute value of the vertex matches the attribute "ntex" and attribute "win_id" in a group, it means that the data structure of the vertex already exists. Accordingly, attribute interpolator 36 may set the data structure of the vertex to the data structure indicated by the attribute "vtx_addr" in the group. For example, if two tiles in a mapped region (e.g., mapped region 611) are adjacent, namely tile #1 and tile #2. After generating the data structures for each vertex of tile #1, the attribute interpolator 36 can determine from the intra-frame cache that the data structures for vertices sharing the same edge between tile #1 and tile #2 have been generated. Therefore, when generating the data structures for each vertex of tile #2, the attribute interpolator 36 can omit the step of generating the data structures for vertices sharing the same edge, thereby speeding up the image processing flow.

[0148] Table 11

[0149]

[0150] In one embodiment, vertex generator 37 can generate a current frame header corresponding to the current frame of the output image based on the combined vertex list. Table 12 shows an example of a frame header, where the attribute "win_desc" indicates the display area in the layout 200 of the output image. When the output image enters the next frame, the current frame header will become the reference frame header for the next frame.

[0151] Table 12

[0152]

[0153] When attribute interpolator 36 generates a data structure for the vertices in the display area (or sub-display area) corresponding to the next frame of the output image, attribute interpolator 36 can determine whether the display area of ​​the next frame matches the display area indicated by the reference frame header. Specifically, the reference frame header can indicate the inter-frame cache in storage medium 120 corresponding to the display area (or sub-display area) of the reference frame, where the inter-frame cache contains attributes "cache_ref_id1" and "cache_ref_id2". If the attribute "id1" of the display area of ​​the next frame is the same as the attribute "cache_ref_id1" indicated by the reference frame header, and the attribute "id2" of the display area of ​​the next frame is the same as the attribute "cache_ref_id2" indicated by the reference frame header, then attribute interpolator 36 can determine that the two display areas match. Otherwise, attribute interpolator 36 can determine that the two display areas do not match.

[0154] The attribute interpolator 36 can copy data from the reference frame header in response to a match between the display area of ​​the next frame and the display area indicated by the reference frame header. This data includes vertices within the display area indicated by the reference frame header and corresponding data structures for those vertices. The copied data is used to generate a combined vertex list corresponding to the next frame.

[0155] Figure 9 A flowchart illustrating an image processing method for generating a layout comprising multiple images is shown according to an embodiment of the present invention, wherein the image processing method may be performed by, for example... Figure 1The image processing apparatus 10 shown is implemented as follows: In step S901, a correction map and a vertex list corresponding to the input image are obtained, wherein the vertex list includes a first vertex on the correction map, a second vertex on the correction map, a first data structure corresponding to the first vertex, and a second data structure corresponding to the second vertex. In step S902, layout information is received to determine a first display area and a second display area in the layout. In step S903, the layout is divided into multiple tiles. In step S904, a first tile located at the boundary between the first display area and the second display area is selected from the multiple tiles, and a first sub-block corresponding to the first display area is cut from the first tile. In step S905, the first sub-block is mapped to the correction map to obtain a mapped area. In step S906, in response to a third vertex in the mapped area corresponding to the first vertex and the second vertex, an interpolation operation is performed on the first data structure and the second data structure to obtain a third data structure corresponding to the third vertex. In step S907, the vertex list is updated according to the third vertex and the third data structure to generate a combined vertex list. In step S908, the input image is mapped to the layout according to the combined vertex list to generate the output image, and the output image is output.

[0156] In summary, the image processing apparatus of the present invention can generate a layout with multiple display areas based on the vertex list of a calibration map according to the user's layout requirements, wherein different display areas can be used to display different images. The image processing apparatus can map the display area to the calibration map in units of tiles to generate mapped areas, and then perform interpolation operations on the mapped areas according to the vertex list of the calibration map to obtain the data structure of each pixel. The image processing apparatus can map the input image to the display areas of the layout according to the data structure of each pixel to generate an output image. For the boundaries or splicing areas of different display areas, the image processing apparatus can further cut the tiles into sub-blocks or primitives so that the boundaries or splicing areas contain more vertices and their data structures. Accordingly, when mapping the input image to the layout according to the vertex list, the image processing apparatus can allocate more computing resources to the boundaries or splicing areas of the display areas. Therefore, the present invention can generate an output image with multiple display areas with optimal efficiency.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing apparatus for generating a layout comprising multiple images, characterized in that, include: The transceiver acquires a calibration map and a vertex list corresponding to the input image, wherein the vertex list includes a first vertex on the calibration map, a second vertex on the calibration map, a first data structure corresponding to the first vertex, and a second data structure corresponding to the second vertex. Storage medium, storing multiple modules; as well as A processor, coupled to the storage medium and the transceiver, accesses and executes the plurality of modules, wherein the plurality of modules includes: A region of interest combiner receives layout information via the transceiver to determine a first display area and a second display area in the layout. The frame tile segmentation unit divides the layout into multiple tiles; The tile separator selects the first tile located at the junction of the first display area and the second display area from the plurality of tiles, and cuts out a first sub-block corresponding to the first display area from the first tile; A coordinate converter maps the first sub-block to the correction map to obtain a mapped region; An attribute interpolator, in response to the fact that a third vertex in the mapping region corresponds to the first vertex and the second vertex, performs an interpolation operation on the first data structure and the second data structure to obtain a third data structure corresponding to the third vertex; A vertex generator updates the vertex list based on the third vertex and the third data structure to generate a combined vertex list; and The image processing module maps the input image to the layout according to the combined vertex list to generate an output image, and outputs the output image through the transceiver.

2. The image processing apparatus according to claim 1, wherein the plurality of modules further comprises: The primitive assembly unit divides the mapping region into multiple primitives according to the size of the mapping region, wherein the multiple primitives include a first primitive having the third vertex.

3. The image processing apparatus according to claim 1, wherein the plurality of modules further comprises: The primitive assembly unit determines whether the mapped region includes a stitching region of multiple original images based on the vertex list, and in response to the mapping region including the stitching region, divides the mapping region into multiple primitives according to the size of the stitching region, wherein the multiple primitives include a first primitive having the third vertex.

4. The image processing apparatus according to claim 2, wherein each of the plurality of primitives is a geometric structure having vertices.

5. The image processing apparatus according to claim 1, wherein the layout information includes a first original display area and a second original display area in the layout, wherein The region of interest combiner responds to the partial overlap between the first original display area and the second original display area by segmenting a plurality of sub-display areas, including the first display area, from the first original display area, wherein the plurality of sub-display areas do not overlap with each other, and each of the plurality of sub-display areas does not overlap with the second original display area.

6. The image processing apparatus according to claim 5, wherein the layout information includes a first priority order corresponding to the first original display area and a second priority order corresponding to the second original display area, wherein The region of interest combiner responds by segmenting the first original display area when the first priority is lower than the second priority.

7. The image processing apparatus according to claim 1, wherein The region of interest combiner determines the projection method based on the layout information, wherein The coordinate converter maps the first sub-block to the correction map according to the projection method to obtain the mapped area.

8. The image processing apparatus according to claim 1, wherein The attribute interpolator determines a first weight corresponding to the first vertex and a second weight corresponding to the second vertex based on the distance between the third vertex and the first vertex in a first direction.

9. The image processing apparatus according to claim 8, wherein The attribute interpolator calculates a first product of the first attribute value and the first weight in the first data structure, and a second product of the second attribute value and the second weight in the second data structure, to generate the third data structure.

10. The image processing apparatus of claim 8, wherein the first data structure includes a first scaling factor and a first blending weight corresponding to the first vertex, and the second data structure includes a second scaling factor and a second blending weight corresponding to the second vertex, wherein The attribute interpolator calculates a first product of the first mixed weight and the first weight to obtain or update a first accumulated value corresponding to the first scaling factor, and calculates a second product of the second mixed weight and the second weight to obtain or update a second accumulated value corresponding to the second scaling factor, wherein The attribute interpolator responds to the first accumulated value being greater than the second accumulated value by selecting the first scaling factor from the first scaling factor and the second scaling factor to produce the third data structure.

11. The image processing apparatus of claim 10, wherein the storage medium further stores a first storage array and a second storage array, wherein After obtaining the first product, the attribute interpolator determines whether the first scaling factor matches a first address in the first storage array and whether the first address stores data. If the data is already stored at the first address, in response to determining that the first scaling factor matches the first address, the attribute interpolator adds the first multiplier to the second address in the second storage array corresponding to the first address, wherein... If the data is already stored at the first address, in response to the determination that the first scaling factor does not match the first address, the attribute interpolator stores the first scaling factor at a third address in the first storage array and adds the first multiplier to a fourth address in the second storage array corresponding to the third address, wherein... If the first address does not store the data, the attribute interpolator stores the first scaling factor in the first address and adds the first multiplier to the second address.

12. The image processing apparatus of claim 10, wherein the storage medium further stores a first storage array and a second storage array, wherein the first scaling factor matches a first address in the first storage array, wherein The attribute interpolator adds the first multiplication accumulation to the second position in the second storage array corresponding to the first address according to the lookup table.

13. The image processing apparatus according to claim 1, wherein The vertex generator generates a current frame header corresponding to the current frame of the output image based on the combined vertex list, wherein the current frame header indicates a third display area in the layout, wherein The attribute interpolator responds to the matching of a fourth display region with the third display region in the next frame of the output image by copying the data in the current frame header to generate a new combined vertex list corresponding to the next frame, wherein the data includes a fourth vertex corresponding to the third display region and a fourth data structure corresponding to the fourth vertex.

14. The image processing apparatus according to claim 1, wherein the input image is an equidistant rectangular projection image.

15. The image processing apparatus according to claim 1, wherein the layout information includes at least: The coordinates, width, and height of the first region of interest corresponding to the source image, and the coordinates, width, and height of the display area corresponding to the target image.

16. An image processing method for generating a layout comprising multiple images, characterized in that, include: Obtain a calibration map and a vertex list corresponding to the input image, wherein the vertex list includes a first vertex on the calibration map, a second vertex on the calibration map, a first data structure corresponding to the first vertex, and a second data structure corresponding to the second vertex; Receive layout information to determine the first display area and the second display area in the layout; The layout is divided into multiple tiles; Select a first tile from the plurality of tiles that is located at the junction of the first display area and the second display area, and cut out a first sub-block corresponding to the first display area from the first tile; Map the first sub-block to the correction map to obtain the mapped region; In response to the third vertex in the mapping region corresponding to the first vertex and the second vertex, an interpolation operation is performed on the first data structure and the second data structure to obtain a third data structure corresponding to the third vertex; Update the vertex list based on the third vertex and the third data structure to generate a combined vertex list; as well as The input image is mapped to the layout according to the combined vertex list to generate an output image, and the output image is output.

17. An image processing apparatus for generating a layout comprising multiple images, characterized in that, include: The processor is configured to execute: Obtain a calibration map and a vertex list corresponding to the input image, wherein the vertex list includes a first vertex on the calibration map, a second vertex on the calibration map, a first data structure corresponding to the first vertex, and a second data structure corresponding to the second vertex; Receive layout information to determine the first display area and the second display area in the layout; The layout is divided into multiple tiles; Select a first tile from the plurality of tiles that is located at the junction of the first display area and the second display area, and cut out a first sub-block corresponding to the first display area from the first tile; Map the first sub-block to the correction map to obtain the mapped region; In response to the third vertex in the mapping region corresponding to the first vertex and the second vertex, an interpolation operation is performed on the first data structure and the second data structure to obtain a third data structure corresponding to the third vertex; Update the vertex list based on the third vertex and the third data structure to generate a combined vertex list; as well as The input image is mapped to the layout according to the combined vertex list to generate an output image, and the output image is output.

Images