Multi-lens image stitching system and method

Abstract

A multi-lens image stitching system and method is provided, wherein a first lens is set to shoot a first area to generate a first original image, a second lens is set to shoot a second area to generate a second original image, and, when an object conforming to a specific object characteristic is found to appear at least in part in an overlapped area where the first area overlapped with the second area, it is ensured that the object is fully shown in a stitched image generated by stitching the first original image and the second original image.

Description

Technical Field

[0001] This invention relates to a multi-lens image stitching system and method, and more particularly to a multi-lens image stitching system and method that ensures that objects located at the overlapping areas of multiple images are all displayed in the stitched image. Prior Technology

[0002] Image stitching technology is already quite common in daily life. Examples include panoramic image capture on mobile phones and surround-view images commonly used in driver assistance systems. In generating a surround-view image, the vehicle's computer first uses multiple lenses to capture images of the area they are responsible for, obtaining multiple corresponding images. Then, by using the overlapping images between the areas captured by each lens, image alignment and blending operations are performed to stitch these images together into a surround-view image.

[0003] However, current technology, when stitching together images from multiple lenses, often fails to display objects that actually exist in overlapping areas of the shooting locations in the final surround-view image due to its inherent computational logic. This design flaw could potentially cause drivers to miss important objects, leading to accidents. Summary of the Invention

[0004] In view of this, one of the objectives of the present invention is to provide a multi-lens image stitching system that, when stitching images provided by multiple lenses, determines whether there is a specific object with preset characteristics at the overlapping area of ​​the shooting area, and ensures that the image of the specific object can be completely displayed in the stitched image when the aforementioned specific object exists at the overlapping area.

[0005] In view of this, one of the objectives of the present invention is to provide a multi-lens image stitching method that can be applied to a system with multiple lenses. When stitching images provided by multiple lenses, the method determines whether there is a specific object with preset characteristics at the overlapping area of ​​the shooting area, and ensures that the specific object can be completely displayed in the stitched image when the aforementioned specific object exists at the overlapping area.

[0006] From one perspective, this invention proposes a multi-lens image stitching system in one embodiment, comprising a first lens, a second lens, an image processor, and an image display. The first lens is adapted to capture a first region to generate a corresponding first original image; the second lens is adapted to capture a second region to generate a corresponding second original image; the image processor is electrically coupled to the first lens and the second lens respectively and acquires the first original image and the second original image, and stitches the first original image and the second original image to generate a stitched image; the image display is electrically coupled to the image processor to receive and display the stitched image. Wherein, the overlapping portions of the first region and the second region form an overlapping region, which corresponds to a first overlapping image in the first original image and to a second overlapping image in the second original image; the image processor determines whether a first object image conforming to a specific object characteristic appears at least partially in the first overlapping image, and when the first object image appears at least partially in the first overlapping image, the portion of the first object image within the coverage area of ​​the stitched image is fully included in the stitched image.

[0007] In one embodiment, the image processor first determines whether a first object image exists in the first original image, and when it finds that a first object image exists in the first original image, it sets a first block in the first original image that covers the first object image, and then determines whether the first object image appears at least partially in the first overlapping image based on the relative position of the first block and the first overlapping image.

[0008] In one embodiment, the image processor further determines whether a second object image that conforms to the aforementioned specific object characteristics appears at least partially in the second overlapping image.

[0009] In one embodiment, the image processor first determines whether there is a second object image in the second original image that conforms to the aforementioned specific object characteristics, and when the second object image is found to exist in the second original image, a second block covering the second object image is set in the second original image, and then the relative position of the second block and the second overlapping image is used to determine whether the second object image appears at least partially in the second overlapping image.

[0010] In one embodiment, when a first object image appears at least partially in a first overlapping image and a second object image appears at least partially in a second overlapping image, the image processor determines whether the first object image and the second object image correspond to the same object, and when the first object image and the second object image correspond to the same object, the portion of the first object image within the coverage area of ​​the stitched image is completely included in the stitched image and the second object image is not included in the stitched image.

[0011] In one embodiment, the image processor sets corresponding weight values ​​for the positions of each pixel in the first original image and the second original image, and multiplies the pixel value of each pixel in the first original image and the second original image by the corresponding weight value, and then sums them to generate a stitched image. Further, when the first object image appears at least partially in the first overlapping image, the image processor changes the weight values ​​corresponding to each pixel of the first object image from zero to non-zero values.

[0012] In one embodiment, the image processor performs image recognition processing on the first original image through an image recognition neural network to obtain the pixel coordinates of the first object image, and determines whether the first object image appears at least partially in the first overlapping image based on these pixel coordinates.

[0013] From another perspective, one embodiment of the present invention proposes a multi-lens image stitching method, which is suitable for use in a system including a first lens, a second lens, and an image processor. The first lens is configured to capture a first region, and the second lens is configured to capture a second region, with the overlapping portions of the first and second regions forming an overlapping region. This multi-lens image stitching method includes: having the first lens capture the first region to generate a corresponding first original image; having the second lens capture the second region to generate a corresponding second original image; having the image processor acquire the first original image and the second original image; having the image processor determine whether a first object image at least partially appears in the first overlapping image, wherein the first object image conforms to specific object characteristics, and the first overlapping image is the portion of the first original image corresponding to the overlapping region; and having the image processor stitch the first original image and the second original image to generate a stitched image, wherein when the first object image at least partially appears in the first overlapping image, the portion of the first object image within the coverage area of ​​the stitched image is entirely included in the stitched image.

[0014] In one embodiment, the aforementioned step of enabling the image processor to determine whether the first object image appears at least partially in the first overlapping image includes: determining whether the first object image exists in the first original image; when the first object image is found to exist in the first original image, setting a first block in the first original image that covers the first object image; and determining whether the first object image appears at least partially in the first overlapping image based on the relative position of the first block and the first overlapping image.

[0015] In one embodiment, the multi-lens image stitching method further includes: having an image processor determine whether a second object image at least partially appears in a second overlapping image, wherein the second object image conforms to the aforementioned specific object characteristics, and the second overlapping image is the one in the second original image that corresponds to the overlapping area.

[0016] In one embodiment, the aforementioned step of having the image processor determine whether the second object image appears at least partially in the second overlapping image includes: determining whether the second object image exists in the second original image; when the existence of the second object image is found in the second original image, setting a second block in the second original image that covers the second object image; and determining whether the second object image appears at least partially in the second overlapping image based on the relative position of the second block and the second overlapping image.

[0017] In one embodiment, the aforementioned step of having the image processor stitch together the first original image and the second original image to generate a stitched image includes: when the first object image appears at least partially in the first overlapping image and the second object image does not exist in the second overlapping image, making the portion of the first object image within the coverage area of ​​the stitched image completely included in the stitched image; and when the first object image appears at least partially in the first overlapping image and the second object image also appears at least partially in the second overlapping image: determining whether the first object image and the second object image correspond to the same object; when the first object image and the second object image do not correspond to the same object, making the portions of the first object image and the second object image within the coverage area of ​​the stitched image completely included in the stitched image; and when the first object image and the second object image correspond to the same object, making the portion of the first object image within the coverage area of ​​the stitched image completely included in the stitched image and preventing the second object image from being included in the stitched image.

[0018] In one embodiment, the image processor sets corresponding weight values ​​for the positions of each pixel in the first original image and the second original image, and multiplies the pixel value of each pixel in the first original image and the second original image by the corresponding weight value, and then sums them to generate a stitched image. Further, when the first object image appears at least partially in the first overlapping image, the image processor changes the weight values ​​corresponding to each pixel of the first object image from zero to non-zero values.

[0019] In one embodiment, the image processor performs image recognition processing on the first original image through an image recognition neural network to obtain the pixel coordinates of the first object image, and determines whether the first object image appears at least partially in the first overlapping image based on these pixel coordinates.

[0020] By employing the above-mentioned technology, the multi-lens image stitching system and method provided by the present invention can determine whether there is a specific object with preset characteristics at the overlapping area of ​​the shooting area when stitching original images provided by multiple lenses, and ensure that the object image generated by shooting this specific object can be included in the stitched image obtained by stitching these original images when the aforementioned specific object exists at the overlapping area. Simple Explanation of the Diagram

[0021] Figure 1 is a schematic diagram of the architecture of a multi-lens image stitching system according to an embodiment of the present invention. Figure 2 is a flowchart of a multi-lens image stitching method according to an embodiment of the present invention. Figure 3 is a detailed flowchart of step S204 according to an embodiment of the present invention. Figure 4A is a schematic diagram of the original image according to an embodiment of the present invention. Figure 4B is a schematic diagram of original image stitching according to an embodiment of the present invention. Figure 5 is a detailed flowchart of step S206 according to an embodiment of the present invention. Implementation

[0022] To ensure that the description is clearly understood by those skilled in the art, it should be noted that the term "electrically coupled" as used in this specification means that electronic signals can be transmitted between two electrically coupled objects. Unless otherwise specified, the transmission of electronic signals can be wired or wireless, and the direction of transmission can be unidirectional or bidirectional. Furthermore, the embodiments described in this specification are merely to enable those skilled in the art to easily understand the invention; the details in these embodiments are not all necessary conditions for implementing the invention.

[0023] Please refer to Figure 1, which is a schematic diagram of the architecture of a multi-lens image stitching system according to an embodiment of the present invention. In this embodiment, the multi-lens image stitching system 10 includes lenses 100 and 102, an image processor 110, and an image display 120. As shown, lens 100 is adapted to capture area 170 to generate a corresponding original image IM1, and lens 102 is adapted to capture area 175 to generate a corresponding original image IM2. The overlapping areas 170 and 175 form an overlapping area 180, indicated by a diagonal line. The image processor 110 is electrically coupled to lenses 100 and 102 to acquire the original images IM1 and IM2 generated by lenses 100 and 102 respectively, and after appropriate judgment, stitches the original images IM1 and IM2 together to generate a corresponding stitched image IM3. The stitched image IM3 is transmitted to the image display 120, which is electrically coupled to the image processor 110, and the image display 120 displays the received stitched image IM3 for subsequent use. In one embodiment, the image area contained in the original image IM1 may be smaller than or equal to the shooting area 170 of lens 100, and the image area contained in the original image IM2 may be smaller than or equal to the shooting area 175 of lens 102; that is, the original image IM1 may be an image cropped from the shooting area 170, and the original image IM2 may be an image cropped from the shooting area 175. It should be noted that although the multi-lens image stitching system 10 in this embodiment only contains two lenses, those skilled in the art will understand that a multi-lens image stitching system can actually contain more lenses, and the technical means provided by this invention can also be applied to multi-lens image stitching systems containing more than two lenses. In one embodiment, the multi-lens image stitching system 10 can be a panoramic image stitching system, which maps each pixel of the original images IM1 and IM2 generated by lenses 100 and 102 to a spatial model of the surrounding space, sets a corresponding weight value (e.g., a value between 1 and 0) according to the position of the pixel, multiplies each pixel value of the original images IM1 and IM2 by the corresponding weight value, and then stitches them together to generate a panoramic image stitched image IM3.

[0024] The following will describe in detail one embodiment of the operation of the multi-lens image stitching system 10 by referring to other diagrams.

[0025] Please refer to Figure 2, which is a flowchart of a multi-lens image stitching method according to an embodiment of the present invention. In this embodiment, in step S200, the original image IM1 is generated in the shooting area 170 of lens 100, and the original image IM2 is generated in the shooting area 175 of lens 102. In different embodiments, lens 100 and lens 102 can continuously shoot, or the image processor 110 or other controller can issue instructions to control lens 100 and lens 102 to perform shooting operations when necessary. These details can be adjusted by those skilled in the art as needed, and the present invention is not limited thereto.

[0026] After the original image IM1 is generated by lens 100 or the original image IM2 is generated by lens 102, the image processor 110 can obtain the original image IM1 or the original image IM2 at an appropriate time (e.g., but not limited to when data is needed for image stitching) (step S202). In one embodiment, the original image IM1 or IM2 is first transmitted and stored in a storage device, and then the image processor 110 retrieves the corresponding image data from the storage device when the original image IM1 or IM2 is needed; in another embodiment, the original image IM1 or IM2 is directly provided to the image processor 110 for immediate subsequent processing. These details can be adjusted by those skilled in the art as needed, and the present invention is not limited thereto.

[0027] After acquiring the original image IM1 or IM2, the image processor 110 can perform a judgment operation on the acquired original image in step S204 to determine whether a preset object image appears at least partially in the portion of the processed original image corresponding to the overlapping region 180 (hereinafter referred to as the overlapping image), and generate a corresponding judgment result. The aforementioned preset object image can be image content that conforms to any preset object characteristics. Taking a vehicle surround view imaging system as an example, the object characteristics can be set to be sufficient for the object detection mechanism to determine whether the object in the original image is a pedestrian or a vehicle, but the present invention is not limited thereto.

[0028] After each original image to be used to generate the stitched image has been judged in step S204 and the corresponding judgment result has been obtained, the image processor 110 can determine which data to use when stitching the image based on these judgment results, and use these data to perform image stitching to generate the stitched image (step S206). The stitched image generated by the above steps will be provided to the image display 120 and displayed on the image display 120 when appropriate (step S208).

[0029] The following examples illustrate the judgment operation performed in step S204 for reference by those skilled in the art. Please refer to Figure 3, which is a detailed flowchart of step S204 according to an embodiment of the present invention. In this embodiment, after obtaining the original image in step S202, the image processor 110 can select one original image from the previously obtained original images that will be used for image stitching and has not yet undergone object position judgment (step S300). Specifically, the image processor 110 can obtain the original images IM1 and IM2 simultaneously or sequentially in step S202, and then perform the process shown in Figure 3 on the original images IM1 and IM2 respectively to obtain the technical effect desired when performing step S204.

[0030] Since the same process must be performed on each original image used for image stitching, to avoid redundant and repetitive descriptions, the following explanation will focus on original image IM1. However, those skilled in the art will understand that the same procedure can be applied to all image data to be stitched with original image IM1, such as original image IM2.

[0031] After selecting the original image IM1 via step S300, the image processor 110 executes step S302 to perform image recognition processing on the original image IM1 using appropriate means (such as an artificial intelligence image recognition neural network), thereby determining whether there is an object image in the original image IM1 that matches the image features (hereinafter referred to as specific object features) of a preset target (such as a pedestrian or a vehicle). When the judgment result of step S302 is negative, that is, there is no object image in the original image IM1 that matches the specific object characteristics, the process will proceed to step S304 to set the logic adopted for the original image IM1 not to be changed during the image stitching process, and the process will proceed to step S206 to prepare for subsequent operations; conversely, when the judgment result of step S302 is positive, that is, there is an object image in the original image IM1 that matches the specific object characteristics, the process will proceed to step S31 to determine whether an object image that matches the specific object characteristics appears at least partly in the overlapping image in the original image IM1, and when the judgment result is positive, the part of this object image that is within the coverage area of ​​the stitched image IM3 will be fully included in the stitched image IM3.

[0032] In step S31, a technician can use various techniques to control the image processor 110 to separately confirm the position of the overlapping image within the original image IM1 (step S310) and the position of each object image found in the original image IM1 that conforms to specific object characteristics (step S312). For example, the position of the overlapping image within the original image IM1 can be, but is not limited to, set together when installing lenses 100 and 102, or determined by comparing the contents of the original images IM1 and IM2 each time; the position of the object image within the original image IM1 can be, but is not limited to, directly using the coordinates of the pixels determined to be object images. These variations do not affect the normal operation of the present invention, and those skilled in the art can use existing image recognition and image positioning techniques to complete these steps.

[0033] Please refer to Figure 4A, which is a schematic diagram of the original image IM1 according to an embodiment of the present invention. As shown in the figure, after the above-described steps, the original image IM1 in this embodiment can be determined to include the overlapping image 410 located to its right and three object images 420, 422 and 424. Next, the image processor 110 can determine whether a portion of each object image 420, 422 or 424 is located within the overlapping image 410 (step S314). The judgment operation required in step S314 can be completed by directly comparing the pixel coordinates of the previously confirmed overlapping image 410 with those of the object images 420, 422, and 424; or, as shown in Figure 4A, a block 430 that can cover the object image 420, a block 432 that can cover the object image 422, and a block 434 that can cover the object image 424 can be set first, and then the relative position of the block 430 with the overlapping image 410 can be used to determine whether at least a part of the object image 420 is located within the overlapping image 410, the relative position of the block 432 with the overlapping image 410 can be used to determine whether at least a part of the object image 422 is located within the overlapping image 410, and the relative position of the block 434 with the overlapping image 410 can be used to determine whether at least a part of the object image 424 is located within the overlapping image 410.

[0034] When the judgment result of step S314 is negative, that is, when object images 420, 422, and 424 are not located within the overlapping image 410, the process will proceed to step S304 to set the image stitching logic adopted for the original image IM1 to remain unchanged during subsequent image stitching, and the process will proceed to step S206 to prepare for subsequent operations. Please refer to Figure 4B, which is a schematic diagram of the stitching of the original image IM1 and the original image IM2 according to an embodiment of the present invention. In one embodiment of the image stitching logic in step S304, the overlapping image 410 can be further divided into three overlapping image sub-regions 411, 412, and 413, wherein overlapping image sub-region 411 is the area in the middle of the overlapping image 410, overlapping image sub-region 412 is the area in the overlapping image 410 that is more biased towards the original image IM1, and overlapping image sub-region 413 is the area in the overlapping image 410 that is more biased towards the original image IM2. When stitching original images IM1 and IM2, pixels in the original image IM1 outside of overlapping sub-regions 411 and 413 (i.e., non-overlapping image 460 and overlapping sub-region 412) can be assigned a weight value of 1, while pixels in the original image IM2 in overlapping sub-region 412 can be assigned a weight value of 0 (i.e., this area only displays the image content of the original image IM1); similarly, pixels in the original image IM2 outside of overlapping sub-regions 411 and 412 (i.e., non-overlapping image 470 and overlapping sub-region 413) can be assigned a weight value of 1, while pixels in the original image IM1 in overlapping sub-region 413 can be assigned a weight value of 0. The weight value is 0 (i.e., this area only displays the image content of the original image IM2); otherwise, the image content in the overlapping image sub-region 411 is generated by multiplying the pixel values ​​of the original images IM1 and IM2 by their corresponding weight values ​​and then adding them together (i.e., this area displays the fusion of the image content of the original images IM1 and IM2). For example, the weight values ​​corresponding to each pixel in the overlapping image sub-region 411 of the original image IM1 can be set to gradually decrease from 1 to 0 from left to right, and the weight values ​​corresponding to each pixel in the overlapping image sub-region 411 of the original image IM2 can be set to gradually decrease from 1 to 0 from right to left, thereby making the image content generated by the addition and fusion smoother, but the present invention is not limited to this. Conversely, when the judgment result of step S314 is yes, that is, when there is a part of any object image in the overlapping image 410, the process will proceed to step S316 to set that at least a part of the object image existing in the overlapping image 410 should be displayed in the subsequent image stitching process.Taking Figure 4A as an example, since at least a portion of object images 422 and 424 exist within the overlapping image 410, the image processor 110 sets object images 422 and 424 as image materials (hereinafter referred to as stitching materials) to be used in subsequent image stitching, and then proceeds to step S206 to prepare for subsequent operations. In an embodiment of the panoramic image stitching system, one embodiment of modifying the image stitching logic in step S316 can be to use the pixel coordinates of object images 422 or 424 to infer which part of the shooting area of ​​which lens object images 422 or 424 are located, and determine whether that part of the shooting area is included in the range of non-zero pixel weight values ​​in the original image IM1 or IM2 (i.e., determine whether object images 422 or 424 are displayed in the fused image content of the original images IM1 and IM2). If the judgment result is negative, the weight values ​​corresponding to each pixel of the object image 422 or 424 can be changed to non-zero values ​​so that the object image 422 or 424 (i.e., the stitching material) can be displayed in the fused image content of the original images IM1 and IM2. In other words, when no part of the object image is within the overlapping image 410, the weight values ​​set for each pixel will not change over time; however, when part of the object image is within the overlapping image 410, the weight values ​​set for each pixel may change over time. For example, as the object image moves in the frame, when it is near the overlapping image 410, the weight values ​​of the pixels at the positions that the object image passes through may be changed from the original zero value (i.e., in the original image stitching logic, this position was not displayed in the fused image content when there was no object image) to a non-zero value (i.e., when the object image passes through this position) and then return to zero value (i.e., after the object image leaves this position, it returns to the image stitching logic when there was no object image). .

[0035] The above operation also applies to the original image IM2 used for stitching with the original image IM1, and the original image IM2 can thus be determined to contain stitching material that is set to be used during image stitching because at least a portion of it is located within the overlapping image of the original image IM2. It is also worth noting that although the overlapping image 410 is separated from other parts of the original image IM1 by a dashed line in Figure 4, this is only to facilitate understanding for those skilled in the art and does not imply that such a dashed line must actually be present in the original image IM1.

[0036] As described above, after all the original images to be used to generate the stitched image have been judged in step S204 (or, for example, steps S300-S316) and obtained the corresponding judgment results, the image processor 110 can determine in step S206 which data to use when stitching the image based on these judgment results and generate the stitched image IM3 accordingly. Please refer to Figure 5, which is a detailed flowchart of step S206 according to an embodiment of the present invention. In this embodiment, the image processor 110 first judges in step S500 whether all the original images to be used for this image stitching operation have been judged in step S204 and thus obtained the corresponding judgment results. When the judgment result of step S500 is negative, the image processor 110 will repeat the judgment operation of step S500 at an appropriate time; conversely, when the judgment result of step S500 is positive (in this embodiment, the original images IM1 and IM2 have been judged in step S204), the process will proceed to step S502 for the next judgment.

[0037] In step S502, the image processor 110 determines whether there is stitching material in at least one original image that should be used for image stitching. If the determination result of step S502 is no, that is, there is no stitching material in any of the original images, the process proceeds to step S508 to directly perform the image stitching operation according to the existing situation; for example, as mentioned above, the corresponding weight values ​​set for each pixel are not changed. Conversely, if the determination result of step S502 is yes, the process proceeds to step S504 for further determination. Taking the previous embodiment as an example, since there are object images 422 and 424 set as stitching material in at least the original image IM1, the determination result of step S502 here will be "yes" and the process will proceed to step S504.

[0038] In step S504, the image processor 110 determines whether there is stitching material in different original images that corresponds to the same object located in the overlapping area 180. For example, as mentioned above, the pixel coordinates of the stitching material are used to infer whether the stitching material is included in the shooting areas of both lens 100 and lens 102. Those skilled in the art can use various existing technologies to complete the determination step required by step S504. These details can be easily completed and are not the focus of this invention, so they will not be described in detail here. When the determination in step S504 is negative, it means that the stitching material contained in each original image corresponds to different physical objects, and the process proceeds to step S508 to perform image stitching operation using existing materials; for example, as mentioned above, the weight values ​​corresponding to each pixel of the stitching material are directly changed to non-zero values ​​before performing image stitching operation. Conversely, when the determination result in step S504 is positive, it means that there are stitching materials belonging to different original images that correspond to the same physical object, and at this time, it is necessary to proceed to step S506 for further processing.

[0039] In step S506, the image processor 110 selects one of multiple stitching materials corresponding to the same physical object but belonging to different original images for use in subsequent image stitching operations. Other stitching materials corresponding to the same physical object but not selected by the image processor 110 will not be included in the final stitched image IM3. For example, when the object image 432 used as stitching material in the original image IM1 corresponds to the same physical object as a stitching material in the original image IM2, the image processor 110 can arbitrarily or according to certain preset rules select one of these two stitching materials to participate in the image stitching operation. In fact, since these stitching materials correspond to the same physical object, regardless of which one is ultimately selected to be retained in the stitched image, it can still indicate that there is an overlapping area 180 for this physical object. Therefore, this approach not only achieves the effect that the present invention wants to provide, but also avoids the ghosting phenomenon caused by repeatedly displaying the same object in the stitched image.

[0040] After step S506 is completed, the process proceeds to step S508, where, for example, as mentioned earlier, the weight values ​​corresponding to each pixel of the stitching material are changed to non-zero values ​​to generate the final stitched image IM3 to be provided to the image display 120 for display. As previously mentioned, the image stitching operation actually involves many layers of work, such as image alignment and image blending. These multi-layered tasks do not all need to be performed in step S506, but although the related adjustments are complex, they should be well known to those skilled in the art and are not the focus of this invention, so they will not be described in detail here.

[0041] It is worth noting that some stitching materials, although selected to be included in the stitched image IM3, may be partially excluded during the image stitching process. For example, when performing panoramic image rectangularization, since the original images IM1 and IM2 generated in the aforementioned description are mainly used to expand the field of view horizontally during image stitching, if there is a height difference between the original images IM1 and IM2 in the vertical direction, they may need to be appropriately cropped to obtain the stitched image IM3. Therefore, referring to Figure 4, the original image IM1 may be cropped along the cropping line 450 during the image stitching process, and only then can the cropped original image IM1 be used to generate the final stitched image IM3. As shown in Figure 4, during the aforementioned cropping process, a portion of the object image 422 will also be cropped. Therefore, if object image 422 is selected as the stitching material to be included in the stitched image IM3, then the corresponding content finally displayed in the stitched image IM3 will be the part of object image 422 that is located within the coverage area of ​​the stitched image (i.e., below the cut line 450).

[0042] Of course, the above-described techniques are not only applicable to processing the original IM2 image, but also to generating surround view images based on images captured by multiple lenses in an automotive surround view system. Those skilled in the art can make simple adjustments based on the above description to meet different application requirements, which will not be elaborated upon here.

[0043] In summary, by employing the above-described technology, the multi-lens image stitching system and method provided by this invention can determine whether a specific object conforming to preset characteristics exists at the overlapping area of ​​the shooting region when stitching original images provided by multiple lenses. Furthermore, when such a specific object exists at the overlapping area, it ensures that the image of the object generated by shooting this specific object can be included in the stitched image obtained after stitching these original images. Therefore, the technology provided by this invention can reduce the problem that objects actually existing at the overlapping area of ​​the shooting region cannot be displayed in the final panoramic image, thereby effectively improving the deficiencies of prior art.

[0044] 10: Multi-lens image stitching system 100, 102: Lens 110: Image Processor 120: Image Display 170, 175: Area 180: Overlapping area 410: Overlapping Images 411, 412, 413: Overlapping image sub-regions 420, 422, 424: Object Images Blocks 430, 432, and 434 450: Cutting line 460, 470: Non-overlapping images IM1, IM2: Original images IM3: Stitched Image S200~S208: Implementation steps of a multi-lens image stitching method according to an embodiment of the present invention S31, S300~S316: Detailed steps of an embodiment of the present invention when performing step S204 S500~S508: Detailed steps of an embodiment of the present invention when performing step S206

Claims

1. A multi-lens image stitching system, characterized in that it comprises: A first lens captures a first region to generate a corresponding first original image; a second lens captures a second region to generate a corresponding second original image; an image processor is electrically coupled to the first lens and the second lens respectively and acquires the first original image and the second original image, the image processor stitches the first original image and the second original image to generate a stitched image; and an image display is electrically coupled to the image processor to receive and display the stitched image, wherein the overlapping portions of the first region and the second region form an overlapping region, the overlapping region corresponds to a first overlapping image in the first original image and corresponds to a second overlapping image in the second original image, the image processor determines whether a first object image that conforms to a specific object characteristic appears at least partially in the first overlapping image, and when the first object image appears at least partially in the first overlapping image, the portion of the first object image within the coverage area of ​​the stitched image is fully included in the stitched image.

2. The multi-lens image stitching system as described in claim 1, wherein the image processor first determines whether the first object image exists in the first original image, and when the first object image is found to exist in the first original image, sets a first block in the first original image that covers the first object image, and then determines whether the first object image appears at least partially in the first overlapping image based on the relative position of the first block and the first overlapping image.

3. The multi-lens image stitching system as claimed in claim 2, wherein the image processor further determines whether a second object image that conforms to the specific object characteristics appears at least partially in the second overlapping image.

4. The multi-lens image stitching system as described in claim 3, wherein the image processor first determines whether there is a second object image in the second original image that conforms to the specific object characteristics, and when the second object image is found to exist in the second original image, sets a second block in the second original image that covers the second object image, and then determines whether the second object image appears at least partially in the second overlapping image based on the relative position of the second block and the second overlapping image.

5. The multi-lens image stitching system as claimed in claim 3, wherein when the first object image appears at least partially in the first overlapping image and the second object image appears at least partially in the second overlapping image, the image processor determines whether the first object image and the second object image correspond to the same object, and when the first object image and the second object image correspond to the same object, the portion of the first object image located within the coverage area of ​​the stitched image is completely included in the stitched image and the second object image is not included in the stitched image.

6. The multi-lens image stitching system as described in claim 1, wherein the image processor sets corresponding weight values ​​based on the position of each pixel in the first original image and the second original image, and multiplies each pixel value of the first original image and the second original image by the corresponding weight value and then adds them together to generate the stitched image.

7. The multi-lens image stitching system as claimed in claim 6, wherein the image processor changes the weight value corresponding to each pixel of the first object image from zero to a non-zero value when the first object image appears at least partially in the first overlapping image.

8. The multi-lens image stitching system as claimed in claim 1, wherein the image processor performs image recognition processing on the first original image through an image recognition neural network to obtain the pixel coordinates of the first object image, and determines whether the first object image appears at least partially in the first overlapping image based on the pixel coordinates.

9. A multi-lens image stitching method, suitable for use in a system including a first lens, a second lens, and an image processor, wherein the first lens is configured to capture a first region, the second lens is configured to capture a second region, and the overlapping portion of the first region and the second region forms an overlapping region, the multi-lens image stitching method being characterized by comprising: The first lens captures the first region to generate a corresponding first original image; the second lens captures the second region to generate a corresponding second original image; the image processor acquires the first original image and the second original image; the image processor determines whether a first object image at least partially appears in a first overlapping image, wherein the first object image conforms to a specific object feature, and the first overlapping image is the portion of the first original image corresponding to the overlapping region; and the image processor stitches the first original image and the second original image to generate a stitched image, wherein when the first object image at least partially appears in the first overlapping image, the portion of the first object image located within the coverage area of ​​the stitched image is entirely included in the stitched image.

10. The multi-lens image stitching method as described in claim 9, wherein the image processor determines whether the first object image at least partially appears in the first overlapping image, comprising: Determine whether the image of the first object exists in the first original image; When the first object image is found to exist in the first original image, a first block covering the first object image is set in the first original image; and the relative position of the first block and the first overlapping image is used to determine whether the first object image appears at least partially in the first overlapping image.

11. The multi-lens image stitching method as described in claim 10 further includes: The image processor determines whether a second object image appears at least partially in a second overlapping image, wherein the second object image conforms to the specific object feature, and the second overlapping image is the one in the second original image that corresponds to the overlapping area.

12. The multi-lens image stitching method as described in claim 11, wherein the image processor determines whether the second object image at least partially appears in the second overlapping image, comprising: Determine whether the second object image exists in the second original image; when the second object image is found to exist in the second original image, set a second block in the second original image that covers the second object image; and determine whether the second object image appears at least partially in the second overlapping image based on the relative position of the second block and the second overlapping image.

13. The multi-lens image stitching method as described in claim 11, wherein causing the image processor to stitch the first original image and the second original image to generate the stitched image includes: When the first object image appears at least partially in the first overlapping image and the second object image does not exist in the second overlapping image, the portion of the first object image within the coverage area of ​​the stitched image is completely included in the stitched image; and when the first object image appears at least partially in the first overlapping image and the second object image also appears at least partially in the second overlapping image: it is determined whether the first object image and the second object image correspond to the same object; when the first object image and the second object image do not correspond to the same object, the portions of both the first object image and the second object image within the coverage area of ​​the stitched image are completely included in the stitched image; and when the first object image and the second object image correspond to the same object, the portion of the first object image within the coverage area of ​​the stitched image is completely included in the stitched image and the second object image is not included in the stitched image.

14. The multi-lens image stitching method as described in claim 9, wherein the image processor sets corresponding weight values ​​based on the positions of each pixel in the first original image and the second original image, and multiplies each pixel value of the first original image and the second original image by the corresponding weight value and then adds them together to generate the stitched image.

15. The multi-lens image stitching method as described in claim 14, wherein the image processor changes the weight value corresponding to each pixel of the first object image from zero to a non-zero value when the first object image appears at least partially in the first overlapping image.

16. The multi-lens image stitching method as described in claim 9, wherein the image processor performs image recognition processing on the first original image through an image recognition neural network to obtain the pixel coordinates of the first object image, and determines whether the first object image appears at least partially in the first overlapping image based on the pixel coordinates.

Images