Device and method for merging candidate areas

Abstract

To provide a method for prompting improved identification and merging of two candidate areas relating to respective two images having an overlapping region.SOLUTION: A first image and a second image have an overlapping region, and at least a portion of a first candidate area and at least a portion of a second candidate area are located in the overlapping region. An image overlap size is determined indicating a size of the overlapping region of the first image and the second image, and a candidate area overlap ratio is determined indicating a ratio of overlap between the first candidate area and the second candidate area. A merging threshold is then determined based on the image overlap size, and, on condition that the candidate area overlap ratio is larger than the merging threshold, the first candidate area and the second candidate area are merged, thereby forming a merged candidate area.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to integrating candidate areas, specifically, integrating a candidate area related to a feature in a first image and a second candidate area related to a feature in a second image.

Background Art

[0002] In some image processing applications, candidate areas within an image are identified, where these candidate areas relate to features within the image, such as multiple objects or multiple segments of an object. For example, such candidate areas may be candidate bounding boxes related to object detection in an image, where these candidate bounding boxes may be generated using an object detection algorithm such as a neural network. In other examples, such candidate areas may be candidate polygons or pixel masks generated using an instance segmentation algorithm, or pixel masks generated from a motion detection algorithm. Once candidate areas in an image are identified, a merging algorithm may be used to determine whether two candidate areas should be merged into a single candidate area. This determination is typically based on a threshold for the overlap ratio between the two candidate areas, i.e., how much they overlap relative to their total size. If the overlap ratio exceeds the threshold, the candidate areas are merged; if the overlap ratio is below the threshold, the candidate areas are not merged. The objective of such a merging algorithm is to address cases where deciding to merge two candidate areas is, in fact, related to the same feature in the image. Such merging algorithms have been developed for scenarios where these candidate areas are identified in a single, consecutive image. However, there are scenarios where such algorithms fail to identify all related situations in which candidate areas should be merged, that is, where the candidate areas are, in fact, related to the same feature in the image.One example of such a scenario is when a candidate area is identified within a non-contiguous image captured by a single image sensor, where two parts of this non-contiguous image (treated as two separate images when identifying the candidate area) have an overlapping region, which in this case is the region captured by the same region of that single image sensor. Another example of such a scenario is when a candidate area is identified within two separate images captured by two separate image sensors, where these two images contain an overlapping region, which in this case represents a portion of the scene captured by both of the two separate image sensors. [Overview of the project]

[0003] An object of the present invention is to facilitate improved identification and integration of two candidate areas related to each of two images, each having an overlapping region.

[0004] According to a first aspect, a method is provided for integrating a first candidate area related to a candidate feature in a first image and a second candidate area related to a candidate feature in a second image. The first image and the second image have overlapping regions, and at least one part of the first candidate area and at least one part of the second candidate area are located in the overlapping region. The method includes determining the size of the overlapping region of the first image and the second image, determining the overlap ratio of the candidate areas, which indicates the overlap ratio between the first candidate area and the second candidate area, determining an integration threshold based on the size of the overlapping region, and integrating the first candidate area and the second candidate area, provided that the overlap ratio of the candidate areas is greater than the integration threshold, thereby forming an integrated candidate area.

[0005] "Candidate features" means an object, a part of an object, a segment of an object, or a portion of an object, which has image motion or any other feature, and which is identified in the previous processing and input into the method relating to the first aspect.

[0006] "Image" in relation to the first and second images means that the first and second images are treated separately when identifying candidate areas. These first and second images may be captured by one first image sensor and one second image sensor, respectively. Alternatively, these first and second images may be the first and second parts of an image captured using a single image sensor, respectively. In the latter case, these first and second images may be transformations of the first and second parts, respectively.

[0007] "Overlapping regions" means, in relation to first and second images having overlapping regions, that a region exists in one of the first and second images that is also included in the other image, either transformed or untransformed, or that a region exists in one of the first and second images that represents the same part of the scene as a region in the other image. In the former case, the first and second images may be transformed versions of each of the wide-angle view regions of the scene, for example, to reduce distortion of feature size relationships. In the latter case, the first and second images may be images captured by first and second image sensors, respectively, which capture their respective, but partially overlapping, parts of the scene.

[0008] "Ratio" refers to any type of normalization relating to the size of the first and second candidate areas, in relation to the ratio of overlap between the first candidate area and the second candidate area.

[0009] The inventors recognized that, with respect to a first candidate area identified in the first image and a second candidate area identified in the second image, the overlap between the first and second candidate areas is limited by the size of the overlapping region. For a candidate feature, where one part is in the first image and another part is in the second image, a common part of that candidate feature will arise, which will occur in the overlapping region; that is, the common part will appear in both the first and second images. This is because the first candidate area related to the candidate feature identified in the first image and the second candidate area related to the candidate feature identified in the second image can only overlap in their overlapping region. To provide this limitation, the integrated threshold according to the first embodiment is determined based on the size of the overlapping region.

[0010] By determining the merge threshold based on the size of overlapping images, the risk of failing to merge a first candidate area and a second candidate area, even if they relate to the same candidate feature, is reduced. Consequently, the risk of the same candidate feature in two separate images being identified as two separate candidate features is reduced.

[0011] The integration threshold may be determined to increase with the size of the overlapping images. For example, the integration threshold may be determined to be proportional to the size of the overlapping images. Thus, for a given overlap size, the integration threshold is determined to be that given overlap size multiplied by a proportionality constant.

[0012] The ratio of overlap between candidate areas may be determined as the intersection of the first and second candidate areas, which is divided by the junction between the first and second candidate areas. This measurement of the overlap ratio of candidate areas is generally called the intersection on the junction (IoU).

[0013] The first candidate area and the second candidate area may be identified by the coordinates of the combined image, which includes the first image and the second image. This may relate, for example, to a case where the first image and the second image are captured by a first image sensor and a second image sensor, respectively, where the first and second image sensors capture different, but partially overlapping, parts of a scene.

[0014] The first image may be a first transformed view of a first portion of a fisheye image, and the second image may be a second transformed view of a second portion of the fisheye image, and the first candidate area and the second candidate area may be identified in the coordinates of the fisheye image or in the coordinates of some other coordinate system. This is the case when the first image and the second image are, respectively, a first and second portion of an image captured using a single image sensor, and here the first image and the second image are transformed versions of the first and second portions, respectively. Therefore, the first candidate area and the second candidate area identified in the first image and the second image, respectively, are inversely transformed so that they are identified in the coordinates of the fisheye image or in the coordinates of some other coordinate system.

[0015] The integrated candidate area may consist of a junction between the first candidate area and the second candidate area. Alternatives are also possible. For example, if the integrated candidate area should have a specific shape, this integrated candidate area may be an area of ​​that specific shape that includes the first candidate area and the second candidate area. For example, the integrated candidate area may be the smallest area of ​​a specific shape that includes the first candidate area and the second candidate area.

[0016] The first and second candidate areas may be one of a bounding box, a pixel mask, and a polygonal area. The merged candidate area may then be of the same type. For example, if the first and second candidate areas are bounding boxes, the merged candidate area may also be a bounding box.

[0017] According to the second embodiment, a non-temporary computer-readable storage medium is provided, which, when executed by a device having processing capabilities, stores instructions for performing the method according to the first embodiment.

[0018] Any additional features of the method relating to the first embodiment described above, if applicable, shall also apply to this second embodiment. To avoid unnecessary repetition, please refer to the above.

[0019] According to a third aspect, a device is provided for integrating a first candidate area related to a candidate feature in a first image and a second candidate area related to a candidate feature in a second image. The first image and the second image have overlapping regions, and at least one portion of the first candidate area and at least one portion of the second candidate area are located in the overlapping region. The device includes a circuit configured to perform: a function for determining the size of the overlapping region of the first image and the second image, which indicates the size of the overlapping region; a function for determining the ratio of the overlapping ratio of the candidate areas, which indicates the overlap between the first candidate area and the second candidate area; a function for determining the integration threshold, which determines the integration threshold based on the size of the overlapping region of the images; and an integration function, which integrates the first candidate area and the second candidate area, provided that the ratio of the overlapping ratio of the candidate areas is greater than the integration threshold, thereby forming an integrated candidate area.

[0020] Any additional features of the method relating to the first embodiment described above, if applicable, shall also apply to this third embodiment. Refer to the above to avoid unnecessary repetition.

[0021] Further scope of the applicability of the present invention will become apparent from the following detailed description. However, while preferred embodiments of the present invention are shown, it should be understood that the detailed description and specific examples are provided for illustrative purposes only, for various changes and modifications within the scope of the present invention will become apparent to those skilled in the art from this detailed description.

[0022] Therefore, it will be understood that the present invention is not limited to specific components of the described devices or specific operations of the described methods, and such devices and methods may vary. It will also be understood that the terms used herein are intended solely to describe specific embodiments and are not intended to limit them. Note that, as used herein and in the appended claims, the articles “a,” “an,” “the,” and “said” are intended to mean that there is one or more elements unless other examples are clearly determined by the context. Thus, for example, a citation such as “a unit” or “the unit” may include several devices, etc. Furthermore, words such as “comprising,” “including,” and “containing” do not exclude other elements or steps.

[0023] The above and other embodiments of the present invention will be described in more detail below with reference to the attached drawings. These drawings should not be considered limiting, but rather are used for illustrative and illustrative purposes. [Brief explanation of the drawing]

[0024] [Figure 1a] This shows a panoramic image captured using a fisheye lens and a single image sensor. [Figure 1b]Shown is what is called a quad view of the image of FIG. 1a, including four transformed views of each of four parts of the image of FIG. 1a. [Figure 1c] Shown is a schematic diagram of two images with overlap. [Figure 2a] Shown are respective diagrams of the first image and the second image, and the first candidate area and the second candidate area, related to candidate features in each of the first image and the second image. [Figure 2b] Shown are further diagrams of the first image and the second image, and the first candidate area and the second candidate area, related to candidate features in each of the first image and the second image. [Figure 3] Shown is a flowchart related to an embodiment of the method of the present disclosure for integrating a first candidate area related to a candidate feature in a first image and a second candidate area related to a candidate feature in a second image. [Figure 4] Shown is a schematic diagram related to an embodiment of the device of the present disclosure for integrating a first candidate area related to a candidate feature in a first image and a second candidate area related to a candidate feature in a second image.

Embodiments for Carrying Out the Invention

[0025] The present invention will be described below with reference to the accompanying drawings, where currently preferred embodiments of the present invention are shown. However, the present invention may be embodied in many different forms and should not be understood as limited to the embodiments shown herein.

[0026] In the following, when we refer to a candidate feature, such a candidate feature may relate to an object, part of an object, segment of an object, or part of an object that has motion in the image or any other feature related to the candidate area being identified in a previous process such as object detection, instance segmentation, panoramic segmentation, or motion-based tasks to find where there is movement in the image.

[0027] Furthermore, in the following, when we refer to an image, such images may be captured by each of the image sensors, or they may relate to each of the parts of a single image captured using a single image sensor. In the latter case, the images may be transformations of each of the parts of the single image.

[0028] Furthermore, in the following, when we refer to an overlapping region or overlap in relation to two images, such overlapping region or overlap is a region in one of the two images that is also included in the other image of the two images, whether transformed or not, or a region in one of the two images that represents the same part of the scene as a region in the other image of the two images. In the former case, the two images may be transformed versions of each of the wide-angle view regions of the scene, for example, to reduce distortion of feature size relationships. In the latter case, the first and second images may be images captured by first and second image sensors, respectively, which capture their respective, but partially overlapping, regions of the scene.

[0029] The present invention is applicable when processing is performed separately with respect to the first and second images in order to identify candidate areas in relation to candidate features in the first image and candidate features in the second image, respectively. For example, this processing may relate to object detection where each candidate area is associated with an object that is a candidate feature. Such object detection may use one-stage neural networks, e.g., You Only Look Once (YOLO) architecture, Single Shot Multibox Detector (SSD) / SSD-Lite, anchor-free architectures, e.g., CenterNet, or Fully Convolutional One-Stage Object Detection (FCOS), and transformer-based networks, e.g., Detection Transformer (DETR). Object detection may further relate to two-stage / multi-stage detectors, e.g., region-based convolutional neural networks (R-CNN), Faster R-CNN, etc. Object detection may further be directionality gradient histogram (HOG) based. For object detection, the output candidate areas may be boxes or other types of polygons. This process may further relate to segmentation, such as instance segmentation (e.g., Mask R-CNN) and panoramic segmentation, where the candidate feature associated with each candidate area is a segment. For segmentation, the output candidate areas may be polygons or pixel masks. This process may further relate to motion-based tasks, where the task is to find where there is movement in an image, where the candidate feature associated with each candidate area is a moving part. The output candidate areas from motion-based tasks may be polygons or pixel masks.

[0030] When the above or other processing is performed, candidate features that are partially placed in overlapping regions typically result in candidate areas in both the first and second images. However, because the overlap between candidate areas in different images is limited by the size of the overlap, using the same merge threshold as when merging candidate areas in the same image will, in some cases, result in not merging candidate boxes even if the candidate areas are related to the same candidate feature.

[0031] The present invention therefore relates to determining when to merge candidate areas. For this purpose, non-maximum suppression (NMS) is often applied. For example, most object detection approaches apply a sliding window to a feature map and assign foreground / background confidence scores based on candidate features computed within that window. Some deep learning detectors, such as single-shot multi-box detectors and fast RCNN detectors, use predefined anchor boxes with different feature layer resolutions. Adjacent windows (anchor boxes) have somewhat similar confidence scores and are considered candidate areas. This leads to many candidate areas (candidate features) for the same candidate object. NMS is used to filter proposals based on intersections over joins (IoU), which can be defined as the intersection of two candidate areas separated by the join of those two candidate areas.

[0032] Each candidate area is compared to all other candidate areas by calculating its IoU (Identity of Use) using all other candidate areas. If the IoU is greater than the integration threshold N, the candidate areas are integrated. One typical integration method is to select the proposal with the highest confidence value.

[0033] In the following, we will describe a first scenario in relation to Figure 1a, which shows a panoramic image 110 captured using a fisheye lens; Figure 1b, which shows what is called a quad view of the image in Figure 1a, including four transformed views (images) 112, 114, 116, and 118, one for each of the four parts of the image 110 in Figure 1a; and Figure 1c, which shows a schematic diagram of two images 120 and 130 having an overlapping region 140.

[0034] As can be seen in Figure 1a, the use of a fisheye lens provides a wide-angle view 110, but this view is distorted such that the relative size of features in this image depends on their location within this view. When processing image 110 as disclosed in Figure 1a to identify candidate areas in relation to candidate features in this image, some features may become relatively smaller due to this distortion, and processing this image may lead to errors in relation to such relatively small features. Therefore, instead of processing image 110 as shown in Figure 1a, the image may be divided into parts, such as four parts. Furthermore, to reduce distortion, each part may be designated as a non-contiguous part of an image or a separate image, for example, the first to fourth images (parts) 112, 114, 116, and 118 in the quad view of Figure 1b, so that the relative size of features does not depend too much on one location in any of the images 112, 114, 116, or 118. Processing images 112, 114, 116, and 118 separately provides candidate areas in relation to candidate features in each of the images 112, 114, 116, and 118. Furthermore, image 110 in Figure 1a is divided into parts, and these parts are converted into separate images to reduce distortion, so that each of the images 112, 114, 116, and 118 contains regions that are also included in the other images of images 112, 114, 116, and 118. Here, as previously shown, one region that is included in two of the images 112, 114, 116, and 118 is called an overlapping region of the two images. A schematic diagram of the first image 120 and the second image 130 having an overlapping region 140 is shown in Figure 1c. The first image 120 and the second image 130 in Figure 1c may correspond to each pair of images in Figure 1b, for example, the first image 112 and the second image 114, the first image 112 and the third image 116, the second image 114 and the fourth image 118, and the third image 116 and the fourth image 118.Please note that Figure 1c only provides a schematic diagram of how a feature 150 in the form of an object may have a first portion that is placed only in the first image 120, a second portion that is placed only in the second image 130, and a third portion that is placed only in the overlapping region 140, which means that it is placed in both the first image 120 and the second image 130. For example, in Figure 1c, the overlapping region 140 between the first image 120 and the second image 130 is rectangular. In practice, the overlapping region of two of the four images 112, 114, 116, and 118 in the quad view of Figure 1b is not typically rectangular. However, each overlapping region of the four images 112, 114, 116, and 118 of the quad view in Figure 1b can be determined based on knowledge of how the four images 112, 114, 116, and 118 of the quad view in Figure 1b were created from, for example, each part of image 110 in Figure 1a that was used in each of the four images 112, 114, 116, and 118 of the quad view in Figure 1b, and from the transformations used.

[0035] To ensure proper processing for identifying candidate areas, the division of the original image into its respective parts and the conversion of each of those parts into separate images should preferably be configured such that no region of the original image appears in any of those separate images. This is not the case with the original image 110 in Figure 1a and its separate images 112, 114, 116, and 118 in Figure 1b.

[0036] It should be noted that the quad view in Figure 1b may also be created and used for other reasons, such as to provide a better overview of a scene when one or more images or videos of that scene are being viewed.

[0037] In the following, a second scenario will be described in relation to Figure 1c, which shows a schematic diagram of two overlapping images. As an alternative to using a wide-angle lens such as a fisheye lens, a panoramic image covering a wide angle can be generated by using two image sensors. In the second scenario, the first image 120 is an image captured by the first image sensor, and the second image 130 is an image captured by the second image sensor. The overlapping region 140 is an area captured by both the first and second image sensors. In contrast to the first scenario, this overlapping region is therefore not related to an area in both the first and second images captured by the same image sensor, and therefore does not contain identical image data. Instead, it will contain image data from the first and second image sensors, respectively, representing the same part of the scene captured by the first and second image sensors. In this second scenario, the first image 120 and the second image 130 may be later joined together to form a single combined image that represents the scene captured by the first image sensor and the second image sensor. However, in this second scenario, it may also be preferable to process the first image 120 and the second image 130 separately when performing a process to identify candidate areas using, for example, one of the types of processing described above. For example, the first image sensor and the second image sensor may be located in the same place, or they may be associated with one of each of two processing units, so that the processing of the first image 120 and the second image 130 can be performed in parallel, and candidate areas as a result of that processing can be provided in a shorter time. Furthermore, processing the first image 120 and the second image 130 separately results in processing two smaller images, which enables the identification of smaller candidate areas, for example, by enabling the detection of smaller objects. There may be hardware limitations that prevent the combination of the first image 120 and the second image 130 from being processed as a single combined image.

[0038] In different scenarios and in different specific implementations, the overlapping region between the two images may be of different sizes, as reflected in Figures 2a and 2b. Figure 2a shows a diagram of a first image 220a and a second image 230a with an overlapping area 240a. Furthermore, in Figure 2a, in the first image 220a and the second image 230a, the first candidate area 260a in bounding box form is shown with a dashed line, and the second candidate area 270a in bounding box form is shown with a dotted line, relating to the candidate feature 250a in object form, respectively. Figure 2b shows a further diagram of a first image 220b and a second image 230b with an overlapping area 240b. Furthermore, in Figure 2b, in the first image 220b and the second image 230b, the first candidate area 260b in the form of a bounding box is shown with a dashed line, and the second candidate area 270b in the form of a bounding box is shown with a dotted line, in relation to the candidate feature 250b in the form of the object, respectively. As can be seen from the figures 2a and 2b, the overlapping region 240a of the first image 220a and the second image 230a in Figure 2a is smaller than the overlapping region 240b of the first image 220b and the second image 230b in Figure 2b. Furthermore, regarding the size of the intersection between a candidate area in one of those images and a candidate area in the other, if those images are limited by the size of their overlapping region, then the size of the intersection between candidate areas for a feature located in the overlapping region will depend on the size of that overlapping region. As can be seen from Figures 2a and 2b, the intersection between candidate area 260a in the first image 220a of Figure 2a and candidate area 270a in the second image 230a is smaller than the intersection between candidate area 260b in the first image 220b of Figure 2b and candidate area 270b in the second image 230b of Figure 2b because the overlapping region 240a in Figure 2a is smaller than the overlapping region in Figure 2b.On the other hand, the size of the junction between a candidate area in one of those images and a candidate area in the other image is generally not significantly affected by the size of the overlapping region, but rather depends on the size of the feature if the candidate areas in those images are related to the same feature, where at least one part of it is located in the overlapping region.

[0039] Figures 2a and 2b each disclose two candidate areas in the form of rectangular bounding boxes, each of which has one part within an overlapping region. The merging of candidate areas in this disclosure is applicable, with modifications as necessary, to other types and shapes of candidate areas, such as pixel masks and polygons, as described herein. Furthermore, the two candidate areas in Figures 2a and 2b relate to features in the form of an object. However, the merging of candidate areas in this disclosure is applicable, with modifications as necessary, to other types of features, such as segments resulting from different types of segmentation, as described herein, and to other types of parts that have motion resulting from motion-based tasks, as described herein.

[0040] Embodiments of a method 300 for integrating first candidate areas 260a, 260b associated with candidate features 250, 250a, 250b in the first images 220, 220a, 220b and second candidate areas 270a, 270b associated with candidate features 250, 250a, 250b in the second images 230, 230a, 230b will be described with reference to Figure 3, which shows a flowchart 300, and with reference to Figures 1a, 1b, 1c, 2a, and 2b. The first images 220, 220a, 220b and the second images 230, 230a, 230b have overlapping regions 240, 240a, 240b, and at least one part of the first candidate areas 260a, 260b and at least one part of the second candidate areas 270a, 270b are located in the overlapping regions 240a, 240b.

[0041] The first images 220, 220a, 220b and the second images 230, 230a, 230b may relate to any of the scenarios that result in them having overlapping regions 240, 240a, 240b.

[0042] For example, as will be explained in relation to Figures 1a and 1b, the first images 220, 220a, and 220b may be first transformed views of a first portion of the fisheye image 110, that is, a panoramic image 110 captured using a fisheye lens, and the second images 230, 230a, and 230b may be second transformed views of a second portion of the fisheye image 110. The first candidate areas 260a and 260b and the second candidate areas 270a and 270b may be identified in this example by the coordinates of image 110. For example, if candidate areas have been identified based on a previous process that identifies candidate features in the first transformed view (image) 112 and the second transformed view (image) 114 of the quad view in Figure 1b, then these candidate areas may first be transformed to the coordinates of image 110 in Figure 1a using the inverse of the transformation used to transform each corresponding part of image 110 into the first transformed view (image) 112 and the second transformed view (image) 114. Therefore, even if the first candidate areas 260a and 260b relate to candidate features 250, 250a, and 250b in the first images 220, 220a, and 220b, and the second candidate areas 270a and 270b relate to candidate features 250, 250a, and 250b in the second images 230, 230a, and 230b, this does not necessarily mean that the first candidate areas 260a and 260b and the second candidate areas 270a and 270b are identified by their respective coordinates in the first images 220, 220a, and 220b and the second images 230, 230a, and 230b. Note that the first candidate areas 260a and 260b and the second candidate areas 270a and 270b in Figures 2a and 2b are bounded by rectangular boxes. However, in the scenarios related to Figures 1a and 1b, such bounding boxes would correspond to the rectangular bounding boxes identified in the four transformed views (images) 112, 114, 116, and 118, transformed to the coordinates of image 110.

[0043] In another example, as disclosed in relation to Figure 1c, the first images 220, 220a, and 220b may be images captured by a first image sensor, and the second images 230, 230a, and 230b may be images captured by a second image sensor. The first candidate areas 260a and 260b and the second candidate areas 270a and 270b may be identified in this example by the coordinates of a combined image including the first images 220, 220a, and 220b and the second images 230, 230a, and 230b.

[0044] The first candidate areas 260a and 260b and the second candidate areas 270a and 270b may be any of the candidate areas obtained from a previous process, such as the type of processing described above, for identifying candidate features in the first image 220, 220a, and 220b and the second image 270a and 270b, respectively. Depending on the scenario, the first candidate areas 260a and 260b and the second candidate areas 270a and 270b may be transformed versions of the candidate areas identified in the first image 220, 220a, and 220b and the second image 230, 230a, and 230b, respectively.

[0045] Furthermore, the first candidate areas 260a and 260b and the second candidate areas 270a and 270b may be candidate areas directly derived from previous processing to identify candidate features in the first images 220, 220a and 220b and the second images 270a and 270b, which may or may not be transformed depending on the scenario. Alternatively, the first candidate areas 260a and 260b and the second candidate areas 270a and 270b may be candidate areas derived from first integrating candidate areas in each image based on a candidate area integration algorithm such as NMS described above. Therefore, for example, the first candidate areas 260a and 260b may relate to two or more candidate areas without considering any of the candidate areas identified in the first images 220, 220a, and 220b, which are integrated using an integration algorithm applied only to the candidate areas in the first images 220, 220a, and 220b.

[0046] The first candidate areas 260a, 260b and the second candidate areas 270a, 270b may be bounding boxes, pixel masks, or polygonal areas as described herein. The shapes of the first candidate areas 260a, 260b and the second candidate areas 270a, 270b depend on the type of processing used to identify them, for example, the types of processing described herein. Furthermore, the shapes of the first candidate areas 260a, 260b and the second candidate areas 270a, 270b also depend on the transformations performed on the corresponding candidate areas identified in the first images 220, 220a, 220b and the second images 230, 230a, 230b, respectively, if the first images 220, 220a, 220b and the second images 230, 230a, 230b are each transformed parts of a single image 110.

[0047] Method 300 determines the size of the overlapping areas 240, 240a, and 240b between the first images 220, 220a, and 220b and the second images 230, 230a, and 230b, in S310. The method for determining the size typically depends on the shape of the overlapping areas. For example, for overlapping areas 240, 240a, and 240b that are rectangular and have a uniform width, as disclosed in Figures 1c, 2a, and 2b, the size of the overlapping areas may be set to the width of the overlap.

[0048] Furthermore, S320 determines the overlap ratio of the candidate areas, which indicates the overlap ratio between the first candidate areas 260a, 260b and the second candidate areas 270a, 270b. Any type of normalization may be used that relates to the size of the first candidate areas 260a, 260b and the second candidate areas 270a, 270b, typically relating to the size of the junction between the first candidate areas 260a, 260b and the second candidate areas 270a, 270b. For example, the overlap ratio of the candidate areas may be determined as the intersection of the first candidate areas 260a, 260b and the second candidate areas 270a, 270b, which are divided by the junction between the first candidate areas 260a, 260b and the second candidate areas 270a, 270b.

[0049] Next, in S330, a merge threshold is determined based on the size of the overlapping images. The merge threshold may be determined to increase with the size of the overlapping images. For example, the merge threshold may be determined to be proportional to the size of the overlapping images. Therefore, for a given overlap size, the merge threshold is determined to be that given overlap size multiplied by a proportionality constant.

[0050] Condition C335 is that the ratio of overlap between candidate areas is greater than the integration threshold, and the first candidate areas 260a and 260b and the second candidate areas 270a and 270b are integrated to form an integrated candidate area S340. The integrated candidate area may consist of the junction between the first candidate areas 260a and 260b and the second candidate areas 270a and 270b.

[0051] By determining the merger threshold based on the size of overlapping images, the risk of failing to merge the first candidate area and the second candidate area, even if they are related to the same feature, is reduced. Consequently, the risk of the same feature in separate images being identified as two separate features is reduced.

[0052] Method 300 may be adapted to provide good representations of different shapes and orientations of features resulting from the identification of different shapes and orientations of candidate areas. For example, if a feature is elongated, different orientations of the feature will result in different proportions of the feature, and therefore, different proportions of the identified candidate areas will typically be in overlapping regions. For example, relating to Figures 2a, 2b, and 2c, if features 250, 250a, and 250c are horizontal rather than vertical, then a smaller proportion of the feature will be in the overlapping regions 240, 240a, and 240b. Therefore, to take this into account, the sizes of the intersection of the overlapping regions 240, 240a, and 240b, and the junction of the first candidate areas 260a, 260b and the second candidate areas 270a, 270b, may be determined. The integration threshold may be determined based on the size of the overlapping area, such that the integration threshold is based on the size of the intersection of the overlapping regions 240, 240a, and 240b, and the size of the junction between the first candidate areas 260a, 260b and the second candidate areas 270a, 270b. For example, the integration threshold may be determined to be proportional to the size of the intersection of the overlapping regions 240, 240a, and 240b, and the size of the junction between the first candidate areas 260a, 260b and the second candidate areas 270a, 270b. Therefore, given the size of the intersection of the overlapping regions 240, 240a, and 240b, and the junction of the first candidate areas 260a, 260b and the second candidate areas 270a, 270b, the integration threshold is determined to be the given size of the intersection of the overlapping regions 240, 240a, and 240b, and the junction of the first candidate areas 260a, 260b and the second candidate areas 270a, 270b, multiplied by a proportionality constant. Such a fit is also suitable when the overlapping regions do not have a uniform width.

[0053] Although embodiments of Method 300 have been described in relation to two images, embodiments of the Method are also applicable when there are three or more images. In such cases, embodiments of Method 300 may be applied to those three or more images in a single step for all of the images, or recursively, that is, first for the first two images that have overlap, then for further images of those three or more images that have overlap with what is derived from the first two images and with either of the two first images, and so on.

[0054] Figure 4 shows a schematic diagram relating to an embodiment of the device 400 of the present disclosure for integrating a first candidate area related to a candidate feature in a first image and a second candidate area related to a candidate feature in a second image, wherein the first and second images have overlapping regions, and at least one portion of the first candidate area and at least one portion of the second candidate area are located in the overlapping region. The device 400 may be, for example, a camera. The device 400 includes a circuit 410, which is configured to perform functions 432, 434, 436, and 438 of the device 400. The circuit 410 may include a processor 412, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 412 is configured to execute program code, which may be configured to perform functions 432, 434, 436, and 438 of the device 400.

[0055] Device 400 may further include memory 430. Memory 430 may be one or more of a buffer, flash memory, hard drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or other suitable devices. In a typical configuration, memory 430 may include non-volatile memory for storing data over long periods and volatile memory that serves as system memory for circuit 410. Memory 430 may exchange data with circuit 410 via a data bus. There may also be associated control lines and address buses between memory 430 and circuit 410.

[0056] The functions 432, 434, 436, and 438 of device 400 may be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on the non-temporary computer-readable medium (memory) 430 of device 400, and may be executed by circuit 410, for example, using the processor 412 within circuit 410. Furthermore, the functions 432, 434, 436, and 438 of device 400 may be standalone software applications or may form part of a software application. The described functions may be considered as methods configured for execution by a processing unit, for example, the processor 412 of circuit 410. Also, while the described functions 432, 434, 436, and 438 may be implemented in software, such functions may also be implemented through dedicated hardware or firmware, or some combination of hardware, firmware, and / or software.

[0057] The circuit 410 is configured to perform an image overlap size determination function 432, which is configured to determine the size of the overlapping area between the first image and the second image, indicating the size of the overlapping region between the first image and the second image.

[0058] The circuit 410 is further configured to perform a function 434 for determining the ratio of overlap between candidate areas, which is configured to determine the ratio of overlap between candidate areas, indicating the overlap between a first candidate area and a second candidate area.

[0059] The circuit 410 is further configured to perform a function 436 for determining an integrated threshold, which is configured to determine an integrated threshold based on the size of overlapping images.

[0060] The circuit 410 is further configured to perform a merging function 438, which is configured to merge a first candidate area and a second candidate area, thereby forming a merged candidate area, provided that the ratio of overlap between the candidate areas is greater than a merging threshold.

[0061] Device 400 and functions 432, 434, 436, and 438 performed by circuit 410 may be further adapted as Method 300 and the corresponding steps of Method 300, respectively, as described in relation to Figure 3.

[0062] Those skilled in the art will understand that the present invention is not limited to the embodiments described above. Rather, many modifications and variations are possible within the scope of the claims. Such modifications and variations can be understood and achieved by those skilled in the art in the implementation of the patented invention by examining the drawings, the disclosure and the appended claims.

Claims

1. 1. A computer-implemented method for merging a first candidate area associated with a candidate feature in a first image and a second candidate area associated with a candidate feature in a second image, the first image and the second image having an overlapping region, and at least a portion of the first candidate area and at least a portion of the second candidate area being located in the overlapping region; determining an image overlap size indicating a size of the overlapping area between the first image and the second image; determining a candidate area overlap ratio indicative of an overlap ratio between the first candidate area and the second candidate area; determining an integration threshold based on a size of the image overlap; merging the first candidate area and the second candidate area, where an overlap ratio of the candidate areas is greater than the merging threshold, thereby forming a merged candidate area; The method includes:

2. The method of claim 1 , wherein the merging threshold is determined to be proportional to a size of the image overlap.

3. 2. The method of claim 1, wherein the candidate area overlap ratio is determined as an intersection of the first candidate area and the second candidate area divided by a union of the first candidate area and the second candidate area.

4. The method of claim 1 , wherein the first candidate area and the second candidate area are identified in coordinates of a combined image that includes the first image and the second image.

5. 2. The method of claim 1, wherein the first image is a first transformed view of a first portion of a fisheye image and the second image is a second transformed view of a second portion of the fisheye image, and the first candidate area and the second candidate area are identified in coordinates of the fisheye image.

6. The method of claim 1 , wherein the combined candidate area comprises a union of the first candidate area and the second candidate area.

7. The method of claim 1 , wherein the first and second candidate areas are one of a bounding box, a pixel mask, and a polygon area.

8. A non-transitory computer readable storage medium having stored thereon instructions which, when executed on a device having processing capabilities, perform the method of any one of claims 1 to 7.

9. 1. A device for merging a first candidate area associated with a candidate feature in a first image and a second candidate area associated with a candidate feature in a second image, the first image and the second image having an overlapping region, and at least a portion of the first candidate area and at least a portion of the second candidate area being located in the overlapping region; The device comprises: an image overlap size determiner configured to determine an image overlap size indicating a size of the overlapping area between the first image and the second image; a candidate area overlap ratio determiner configured to determine a candidate area overlap ratio indicative of an overlap between the first candidate area and the second candidate area; a merger threshold determining function configured to determine a merger threshold based on a size of the overlap of the images; a merging function configured to merge the first candidate area and the second candidate area, if an overlap ratio of the candidate areas is greater than the merging threshold, thereby forming a merged candidate area; A device comprising circuitry configured to perform the steps of:

10. The device of claim 9 , wherein the function of determining the merging threshold is configured to determine the merging threshold to be proportional to a size of the overlap of the images.

11. 10. The device of claim 9, wherein the function for determining an overlap ratio of the candidate areas is configured to determine the overlap ratio of the candidate areas as an intersection of the first candidate area and the second candidate area divided by a union of the first candidate area and the second candidate area.

12. The device of claim 9 , wherein the first candidate area and the second candidate area are identified in coordinates of a combined image including the first image and the second image.

13. 10. The device of claim 9, wherein the first image is a first transformed view of a first portion of a fisheye image and the second image is a second transformed view of a second portion of the fisheye image, and the first candidate area and the second candidate area are identified in coordinates of the fisheye image.

14. The device of claim 9 , wherein the combined candidate area comprises a union of the first candidate area and the second candidate area.

15. The device of claim 9 , wherein the first and second candidate areas are one of a bounding box, a pixel mask, and a polygonal area.

Images