DEVICE AND METHOD FOR CONSOLIDATING CANDIDATE AREAS - Patent application

Abstract

To provide a method for prompting improved identification and merging of two candidate areas relating to respective two images having an overlapping region.SOLUTION: A first image and a second image have an overlapping region, and at least a portion of a first candidate area and at least a portion of a second candidate area are located in the overlapping region. An image overlap size is determined indicating a size of the overlapping region of the first image and the second image, and a candidate area overlap ratio is determined indicating a ratio of overlap between the first candidate area and the second candidate area. A merging threshold is then determined based on the image overlap size, and, on condition that the candidate area overlap ratio is larger than the merging threshold, the first candidate area and the second candidate area are merged, thereby forming a merged candidate area.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to merging candidate areas, and in particular to merging a candidate area associated with a feature in a first image with a second candidate area associated with a feature in a second image. [Background technology]

[0002] In some image processing applications, candidate areas in an image are identified, where the candidate areas are associated with features in the image, such as objects or segments of an object. For example, such candidate areas may be candidate bounding boxes associated with object detection in the image, where the candidate bounding boxes may be generated using an object detection algorithm using a neural network or the like. In other examples, such candidate areas may be candidate polygons or pixel masks generated using an instance segmentation algorithm, or pixel masks generated from a motion detection algorithm. Once candidate areas in an image are identified, a merging algorithm may be performed to determine whether two candidate areas should be merged into a single candidate area. Such a determination is typically based on a threshold for the overlap ratio between two candidate areas, i.e., how much the candidate areas overlap in relation to their total size. For overlap ratios above the threshold, the candidate areas are merged, and for overlap ratios below the threshold, the candidate areas are not merged. The aim of such a merging algorithm is that the decision to merge two candidate areas should correspond to cases where the two candidate areas are in fact associated with the same feature in the image. Such merging algorithms have been developed for scenarios where the candidate areas are identified in a single consecutive image. However, there are scenarios where such algorithms are not successful in identifying all relevant situations where the candidate areas should be merged, i.e., the candidate areas are in fact associated with the same feature in the image.One example of such a scenario is when a candidate area is identified in non-consecutive images captured by a single image sensor, where two portions of the non-consecutive images (which are treated as two separate images in identifying the candidate area) have overlapping regions, in this case imaged by the same region of the single image sensor. Another example of such a scenario is when a candidate area is identified in two separate images captured by two separate image sensors, where the two images contain overlapping regions, in this case representing a portion of the scene imaged by both of the two separate image sensors. Summary of the Invention

[0003] An object of the present invention is to facilitate improved identification and integration of two candidate areas associated with two respective images, the two respective images having overlapping regions.

[0004] According to a first aspect, there is provided a method for merging a first candidate area associated with a candidate feature in a first image and a second candidate area associated with a candidate feature in a second image, the first image and the second image having an overlapping region, and at least a portion of the first candidate area and at least a portion of the second candidate area being located in the overlapping region, the method including: determining an image overlap size indicative of a size of the overlapping region of the first image and the second image; determining a candidate area overlap ratio indicative of an overlap ratio between the first candidate area and the second candidate area; determining a merging threshold based on the image overlap size; and merging the first candidate area and the second candidate area if the candidate area overlap ratio is greater than the merging threshold, thereby forming a merged candidate area.

[0005] By "candidate feature" it is meant an object, part of an object, segment of an object, area of ​​an object having image motion or any other feature that has been identified in a previous process and that is input into the method of the first aspect.

[0006] "Image" refers to a first image and a second image, meaning that the first image and the second image are treated separately in identifying candidate areas. The first image and the second image may be captured by one of the first image sensor and one of the second image sensor, respectively. Alternatively, the first image and the second image may be a first portion and a second portion, respectively, of an image captured using a single image sensor. In the latter case, the first image and the second image may be a transformation of the first portion and the second portion, respectively.

[0007] "Overlapping region," in reference to first and second images having overlapping regions, means that there is a region in one of the first and second images that is also included in the other image, transformed or untransformed, or that there is a region in one of the first and second images that represents the same portion of a scene as a region in the other image. In the former case, the first and second images may be respective transformed portions of wide-angle views of the scene, e.g., to reduce distortions in feature size relationships. In the latter case, the first and second images may be images captured by first and second image sensors, respectively, that capture respective but partially overlapping portions of the scene.

[0008] "Ratio" means any kind of normalization related to the size of a first and second candidate area in relation to the ratio of overlap between these first and second candidate areas.

[0009] The inventors have recognized that for a first image and a second image having an overlap region, a first candidate area identified in the first image and a second candidate area identified in the second image, the overlap between the first candidate area and the second candidate area is limited in size by the size of the overlap region. For a candidate feature having a portion in the first image and a portion in the second image, a common portion of the candidate feature will occur in the overlap region, i.e., the common portion will appear in both the first image and the second image, because a first candidate area associated with the candidate feature identified in the first image and a second candidate area associated with the candidate feature identified in the second image can only overlap in the overlap region. To provide this limitation, the integration threshold according to the first aspect is determined based on the size of the overlap region.

[0010] By determining the merging threshold based on the size of image overlap, the risk of not merging a first candidate area and a second candidate area even though they are associated with the same candidate feature is reduced, thus reducing the risk that the same candidate feature in two separate images is identified as two separate candidate features.

[0011] The merging threshold may be determined to increase with the size of the image overlap. For example, the merging threshold may be determined to be proportional to the size of the image overlap. Thus, for a given overlap size, the merging threshold is determined to be that given overlap size multiplied with a proportionality constant.

[0012] The overlap ratio of the candidate areas may be determined as the intersection of the first candidate area and the second candidate area divided by the union of the first candidate area and the second candidate area, this measure of the overlap ratio of the candidate areas is commonly referred to as the intersection over union (IoU).

[0013] The first candidate area and the second candidate area may be identified in coordinates of a combined image including the first image and the second image, which may be relevant, for example, when the first image and the second image are captured by a first image sensor and a second image sensor, respectively, where the first image sensor and the second image sensor capture respective but partially overlapping portions of a scene.

[0014] The first image may be a first transformed view of a first portion of a fisheye image, and the second image may be a second transformed view of a second portion of the fisheye image, and the first and second candidate areas may be identified in the coordinates of the fisheye image or in some other coordinate system. This is the case when the first and second images are respectively the first and second portions of an image captured using a single image sensor, where the first and second images are respectively transformed versions of the first and second portions. Thus, the first and second candidate areas identified in each of the first and second images are inversely transformed so that they are identified in the coordinates of the fisheye image or in some other coordinate system.

[0015] The combined candidate area may consist of a union of the first candidate area and the second candidate area. Alternatives are contemplated. For example, if the combined candidate area is to have a particular shape, the combined candidate area may be an area of ​​the particular shape that includes the first candidate area and the second candidate area. For example, the combined candidate area may be the smallest area of ​​the particular shape that includes the first candidate area and the second candidate area.

[0016] The first and second candidate areas may be one of a bounding box, a pixel mask, and a polygon area. The combined candidate area may then be of the same type. For example, if the first and second candidate areas are bounding boxes, the combined candidate area may also be a bounding box.

[0017] According to a second aspect, there is provided a non-transitory computer readable storage medium having stored thereon instructions which, when executed by a device having processing capability, perform the method according to the first aspect.

[0018] Any additional features described above for the method according to the first aspect, where applicable, also apply to this second aspect: in order to avoid unnecessary repetition, please refer to the above.

[0019] According to a third aspect, there is provided a device for merging a first candidate area associated with a candidate feature in a first image and a second candidate area associated with a candidate feature in a second image, the first image and the second image having an overlapping region, and at least a portion of the first candidate area and at least a portion of the second candidate area being located in the overlapping region, the device including circuitry configured to perform an image overlap size determining function configured to determine an image overlap size indicative of a size of the overlapping region of the first image and the second image, a candidate area overlap ratio determining function configured to determine a candidate area overlap ratio indicative of an overlap between the first candidate area and the second candidate area, a merging threshold determining function configured to determine a merging threshold based on the image overlap size, and a merging function configured to merge the first candidate area and the second candidate area if the candidate area overlap ratio is greater than the merging threshold, thereby forming a merged candidate area.

[0020] Any additional features described above for the method according to the first aspect, where applicable, also apply to this third aspect: in order to avoid unnecessary repetition, please refer to the above.

[0021] Further scope of applicability of the present invention will become apparent from the following detailed description. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the present invention, are given for purposes of illustration only, since various changes and modifications within the scope of the invention will become apparent to those skilled in the art from this detailed description.

[0022] Thus, it is to be understood that the invention is not limited to the particular components of the described devices or the particular operation of the described methods, as such devices and methods may vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. It should be noted that, as used in this specification and the appended claims, the articles "a," "an," "the," and "said" are intended to mean that there are one or more elements, unless the context clearly dictates otherwise. Thus, for example, reference to "a unit" or "the unit" may include several devices, and the like. Further, the words "comprising," "including," "containing," and the like do not exclude other elements or steps.

[0023] These and other aspects of the present invention will now be described in more detail with reference to the accompanying drawings, which should not be considered limiting, but are instead used for explanation and understanding. [Brief description of the drawings]

[0024] [Figure 1a] 1 shows a panoramic image captured using a fisheye lens and a single image sensor. [Figure 1b]1 shows what is called a quad view of the image of FIG. 1a, which includes four transformed views, one each of the four regions of the image of FIG. 1a. [Figure 1c] 1 shows a schematic diagram of two images with overlap. [Figure 2a] 1 shows diagrams of a first image and a second image, and a first candidate area and a second candidate area associated with candidate features in the first image and the second image, respectively; [Figure 2b] FIG. 2 shows further views of the first and second images and first and second candidate areas associated with candidate features in the first and second images, respectively; [Diagram 3] 1 shows a flowchart relating to an embodiment of a method of the present disclosure for merging a first candidate area associated with a candidate feature in a first image and a second candidate area associated with a candidate feature in a second image. [Figure 4] FIG. 1 shows a schematic diagram relating to an embodiment of a device of the present disclosure that merges a first candidate area associated with a candidate feature in a first image and a second candidate area associated with a candidate feature in a second image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The present invention will now be described with reference to the accompanying drawings, in which there is shown a presently preferred embodiment of the invention. The invention may, however, be embodied in many different forms and should not be construed as limited to the embodiment shown.

[0026] In the following, when referring to candidate features, such candidate features may relate to an object, part of an object, segment of an object, area where there is image motion or any other feature related to which the candidate area has been identified in previous processing, such as object detection, instance segmentation, panoramic segmentation, or motion-based tasks to find where there is motion in an image.

[0027] Furthermore, in the following, when referring to images, such images may be captured by each one of the image sensors, or may relate to each of the portions of a single image captured using a single image sensor, in which case the images may be transforms of each of the portions of the single image.

[0028] Furthermore, in the following, when reference is made to an overlap region or overlap in relation to two images, such an overlap region or overlap is a region in one of the two images that is also included in the other of the two images, either transformed or not, or that represents the same portion of the scene as a region in the other of the two images. In the former case, the two images may be respective transformed portions of a wide-angle view of the scene, for example to reduce distortions in feature size relationships. In the latter case, the first and second images may be images captured by first and second image sensors, respectively, where the first and second image sensors capture respective but partially overlapping portions of the scene.

[0029] The present invention is applicable when processing is performed separately in relation to the first image and the second image to identify candidate areas in relation to candidate features in the first image and candidate features in the second image, respectively. For example, the processing may relate to object detection where the candidate features to which each candidate area is associated are objects. Such object detection may use one-stage neural networks, e.g., You-Only-Look-Once (YOLO) architecture, Single-Shot Multibox Detector (SSD) / SSD-Lite, anchor-free architectures, e.g., CenterNet, or fully convolutional one-stage object detection (FCOS), and transformer-based networks, e.g., DEtection TRansformer (DETR). Object detection may further relate to two-stage / multi-stage detectors, e.g., Region-Based Convolutional Neural Network (R-CNN), Faster R-CNN, etc. Object detection may further be Histogram of Oriented Gradients (HOG) based. For object detection, the output candidate areas may be boxes or other types of polygons. This process may be further related to segmentation, such as instance segmentation (e.g., Mask R-CNN) and panoramic segmentation, where the candidate feature associated with each candidate area is a segment. For segmentation, the output candidate areas may be polygons or pixel masks. This process may be further related to motion-based tasks, where the task is to find where there is movement in an image, where the candidate feature associated with each candidate area is a part where there is movement. The output candidate areas from motion-based tasks may be polygons or pixel masks.

[0030] Once this or other processing has been performed, a candidate feature that lies partially in an overlap region will typically result in a candidate area in both the first image and the second image, but because the overlap between candidate areas in different images is limited by the size of the overlap, using the same merging threshold as when merging candidate areas in the same image will in some cases result in not merging candidate boxes even if the candidate areas are associated with the same candidate feature.

[0031] The present invention is therefore concerned with deciding when to merge candidate areas. For this purpose, non-maximum suppression (NMS) is often applied. For example, most object detection approaches apply a sliding window on the feature map and assign foreground / background confidence values ​​(scores) depending on the candidate features calculated within the window. Some deep learning detectors, such as single-shot multibox detectors and Fast RCNN detectors, use predefined anchor boxes with different feature layer resolutions. Adjacent windows (anchor boxes) have somewhat similar confidence values ​​and are considered as candidate areas. This leads to many candidate areas (candidate features) for the same candidate object. NMS is used to filter proposals based on intersection over union (IoU), which can be defined as the intersection of two candidate areas divided by the union of the two candidate areas.

[0032] Each candidate area is compared with all other candidate areas by calculating the IoU of this candidate area with all other candidate areas. If the IoU is greater than a consolidation threshold N, the candidate area is consolidated. One typical consolidation method is to select the proposal with the highest confidence value.

[0033] In the following, the first scenario is described in relation to FIG. 1a showing a panoramic image 110 captured using a fisheye lens, FIG. 1b showing what is called a quad view of the image of FIG. 1a, including four transformed views (images) 112, 114, 116, 118, one each of the four parts of the image 110 of FIG. 1a, and FIG. 1c showing a schematic diagram of two images 120, 130 having an overlapping region 140.

[0034] As can be seen from FIG. 1a, the use of a fisheye lens provides a wide-angle view 110, but the view is distorted such that the relative size of features in the image depends on the location in the view. When processing the image 110 as disclosed in FIG. 1a to identify candidate areas associated with candidate features in the image, some features may be relatively small due to the distortion, and processing the image may introduce errors associated with such relatively small features. Therefore, instead of processing the image 110 as shown in FIG. 1a, the image may be divided into regions, such as four regions. Further, to reduce distortion, each region may be converted into what may be designated as a non-contiguous image portion or separate image, for example, the first through fourth images (portions) 112, 114, 116, 118 in the quad view of FIG. 1b, such that the relative size of features is less dependent on the location in one of the images 112, 114, 116, 118. Processing the images 112, 114, 116, 118 separately provides candidate areas associated with candidate features in each of the images 112, 114, 116, 118. Furthermore, the image 110 of FIG. 1a is divided into each of the regions, which are converted into separate images, so as to reduce distortion and so that each of the images 112, 114, 116, 118 contains regions that are also contained in the other images 112, 114, 116, 118. As previously indicated herein, a region that is contained in two of the images 112, 114, 116, 118 is called an overlap region of the two images. A schematic diagram of a first image 120 and a second image 130 with an overlap region 140 is shown in FIG. 1c. The first image 120 and the second image 130 in FIG. 1c may correspond to each pair of images in FIG. 1b, for example, the first image 112 and the second image 114, the first image 112 and the third image 116, the second image 114 and the fourth image 118, and the third image 116 and the fourth image 118.It should be noted that Fig. 1c only provides a schematic diagram of how a feature 150 in the form of an object may have a first portion located only in the first image 120, a second portion located only in the second image 130, and a third portion located only in the overlapping region 140, meaning located in both the first image 120 and the second image 130. For example, the overlapping region 140 between the first image 120 and the second image 130 in Fig. 1c is rectangular. In a real case, the overlapping region of two images out of the four images 112, 114, 116, 118 of the quad view of Fig. 1b is typically not rectangular. However, each overlapping region in each of the four images 112, 114, 116, 118 of the quad view of FIG. 1b can be determined based on knowledge of how the four images 112, 114, 116, 118 of the quad view of FIG. 1b were created from image 110 of FIG. 1a, e.g., from each portion of image 110 of FIG. 1a and from the transformations used for each of the four images 112, 114, 116, 118 of the quad view of FIG. 1b.

[0035] To ensure proper processing of identifying candidate areas, the division of the original image into each of its parts and the conversion of each of those parts into separate images should preferably be configured such that no region of the original image appears in any one of the separate images, which is not the case with respect to the original image 110 of Figure 1a and the separate images 112, 114, 116, 118 of Figure 1b.

[0036] Note that the quad view of FIG. 1b may also be created and used for other reasons, such as to provide a better overview of a scene when one or more images or videos of the scene are viewed.

[0037] In the following, the second scenario will be described in relation to FIG. 1c, which shows a schematic diagram of two images with overlap. As an alternative to using a wide-angle lens, such as a fisheye lens, a panoramic image covering a wide angle can be generated by using two image sensors. In the second scenario, the first image 120 will be an image captured by the first image sensor, and the second image 130 will be an image captured by the second image sensor. The overlapping area 140 will be an area captured by both the first and second image sensors. In contrast to the first scenario, this overlapping area will therefore not relate to an area in both the first and second images captured by the same image sensor, and therefore will not contain identical image data. Instead, it will contain image data from the first and second image sensors, representing the same part of the scene captured by the first and second image sensors. In this second scenario, the first image 120 and the second image 130 may later be stitched together to form a single combined image showing a representation of the scene captured by the first image sensor and the second image sensor. However, also in this second scenario, it may be advantageous to process the first image 120 and the second image 130 separately when processing to identify candidate areas, for example using one of the types of processing as described herein above. For example, the first image sensor and the second image sensor may be co-located or otherwise associated with each one of two processing units, so that the processing of the first image 120 and the second image 130 may be performed in parallel to provide the resulting candidate areas in a shorter time. Also, processing the first image 120 and the second image 130 separately may result in processing two smaller images, which may allow for the identification of smaller candidate areas, for example by allowing for the detection of smaller objects. There may be hardware limitations that prevent the combination of first image 120 and second image 130 from being processed as one combined image.

[0038] In different scenarios and in different specific implementations, the overlapping area between two images may be of different sizes, as reflected in Fig. 2a and Fig. 2b. Fig. 2a shows a diagram of a first image 220a and a second image 230a with an overlapping area 240a. Furthermore, in Fig. 2a, a first candidate area 260a in the form of a bounding box is shown in dashed lines and a second candidate area 270a in the form of a bounding box is shown in dotted lines in relation to a candidate feature 250a in the form of an object in the first image 220a and in the second image 230a, respectively. Fig. 2b shows a further diagram of a first image 220b and a second image 230b with an overlapping area 240b. Furthermore, in Fig. 2b, a first candidate area 260b in the form of a bounding box is shown in dashed lines and a second candidate area 270b in the form of a bounding box is shown in dotted lines in relation to a candidate feature 250b in the form of an object in the first image 220b and in the second image 230b, respectively. As can be seen from the diagrams in Fig. 2a and 2b, the overlap area 240a of the first image 220a and the second image 230a in Fig. 2a is smaller than the overlap area 240b of the first image 220b and the second image 230b in Fig. 2b. Furthermore, if the images are limited by the size of the overlap area, the size of the intersection between the candidate areas in one of the images and the candidate areas in the other one will depend on the size of the overlap area. As can be seen from the diagrams of Figures 2a and 2b, the intersection between candidate area 260a in the first image 220a and candidate area 270a in the second image 230a in Figure 2a is smaller than the intersection between candidate area 260b in the first image 220b and candidate area 270b in the second image 230b in Figure 2b because the overlapping area 240a in Figure 2a is smaller than the overlapping area in Figure 2b.On the other hand, the size of the connection between a candidate area in one of the images and a candidate area in the other of the images is generally not significantly affected by the size of the overlapping region, but rather depends on the size of the feature if the candidate areas in the images are in fact associated with the same feature, at least in part located in the overlapping region.

[0039] 2a and 2b each disclose two candidate areas in the form of rectangular bounding boxes, each of which has a portion within the overlapping region. The candidate area merging of the present disclosure is applicable mutatis mutandis to other types and shapes of candidate areas, such as pixel masks and polygonal shapes, as described herein above. Furthermore, the two candidate areas of Fig. 2a and 2b are associated with features in the form of objects. However, the candidate area merging of the present disclosure is applicable mutatis mutandis to other types of candidate areas that are the result of other types of processing, associated with other types of features, such as segments resulting from different types of segmentation, as described herein above, and portions with motion resulting from motion-based tasks, as described herein above.

[0040] An embodiment of a method 300 for merging a first candidate area 260a, 260b associated with a candidate feature 250, 250a, 250b in a first image 220, 220a, 220b and a second candidate area 270a, 270b associated with a candidate feature 250, 250a, 250b in a second image 230, 230a, 230b will be described in relation to FIG. 3 showing a flowchart 300 and with reference to FIGS. 1a, 1b, 1c, 2a and 2b. The first image 220, 220a, 220b and the second image 230, 230a, 230b have overlapping regions 240, 240a, 240b, and at least a portion of the first candidate areas 260a, 260b and at least a portion of the second candidate areas 270a, 270b are located in the overlapping regions 240a, 240b.

[0041] The first image 220, 220a, 220b and the second image 230, 230a, 230b may relate to any scenario that results in them having overlapping regions 240, 240a, 240b.

[0042] 1a and 1b, the first image 220, 220a, 220b may be a first transformed view of a first portion of the fisheye image 110, i.e., a panoramic image 110 captured using a fisheye lens, and the second image 230, 230a, 230b may be a second transformed view of a second portion of the fisheye image 110. The first candidate areas 260a, 260b and the second candidate areas 270a, 270b may be identified in coordinates of the image 110, in this example. For example, if candidate areas have been identified based on previous processing that identifies candidate features in the first transformed view (image) 112 and the second transformed view (image) 114 of the quad view of FIG. 1b, then these candidate areas may first be transformed into coordinates in image 110 of FIG. 1a using the inverse of the transformation used to transform each of the corresponding portions of image 110 into the first transformed view (image) 112 and the second transformed view (image) 114. Thus, even if a first candidate area 260a, 260b is associated with a candidate feature 250, 250a, 250b in a first image 220, 220a, 220b and a second candidate area 270a, 270b is associated with a candidate feature 250, 250a, 250b in a second image 230, 230a, 230b, this does not necessarily mean that the first candidate area 260a, 260b and the second candidate area 270a, 270b are identified by their respective coordinates in the first image 220, 220a, 220b and the second image 230, 230a, 230b. It should be noted that the first candidate area 260a, 260b and the second candidate area 270a, 270b in Figures 2a and 2b are rectangular bounding boxes. However, in the scenario associated with Figures 1a and 1b, such bounding boxes would relate to rectangular bounding boxes identified in the four transformed views (images) 112, 114, 116, 118 transformed into the coordinates of image 110.

[0043] As another example, the first image 220, 220a, 220b may be an image captured by a first image sensor and the second image 230, 230a, 230b may be an image captured by a second image sensor, as disclosed in connection with Fig. 1c. The first candidate areas 260a, 260b and the second candidate areas 270a, 270b may be identified in coordinates of a combined image that includes, in this example, the first image 220, 220a, 220b and the second image 230, 230a, 230b.

[0044] The first candidate area 260a, 260b and the second candidate area 270a, 270b may be any candidate area resulting from previous processing to identify candidate features in the first image 220, 220a, 220b and the second image 270a, 270b, respectively, such as the types of processing described herein above. Depending on the scenario, the first candidate area 260a, 260b and the second candidate area 270a, 270b may be a transformation of the candidate areas identified in the first image 220, 220a, 220b and the second image 230, 230a, 230b, respectively.

[0045] Furthermore, the first candidate area 260a, 260b and the second candidate area 270a, 270b may be candidate areas resulting directly from a previous process to identify candidate features in the first image 220, 220a, 220b and the second image 270a, 270b, transformed or not depending on the scenario. Alternatively, the first candidate area 260a, 260b and the second candidate area 270a, 270b may be candidate areas resulting from first merging candidate areas in the respective images based on a candidate area merging algorithm, such as NMS as described herein above. Thus, for example, a first candidate area 260a, 260b may relate to two or more candidate areas identified in the first image 220, 220a, 220b that have been merged using a merging algorithm applied only to the candidate areas in the first image 220, 220a, 220b, i.e., without considering any candidate areas identified in the second image 230, 230a, 230b.

[0046] The first candidate areas 260a, 260b and the second candidate areas 270a, 270b may be bounding boxes, pixel masks, or polygonal areas as described herein above. The shapes of the first candidate areas 260a, 260b and the second candidate areas 270a, 270b depend on the type of processing used to identify them, such as the types of processing described herein above. Furthermore, the shapes of the first candidate areas 260a, 260b and the second candidate areas 270a, 270b also depend on the transformations made to the corresponding candidate areas identified in the first image 220, 220a, 220b and the second image 230, 230a, 230b, respectively, if the first image 220, 220a, 220b and the second image 230, 230a, 230b are transformed portions of a single image 110, respectively.

[0047] In the method 300, an image overlap size is determined S310, indicating the size of the overlap area 240, 240a, 240b between the first image 220, 220a, 220b and the second image 230, 230a, 230b. The manner in which the size is determined will typically depend on the shape of the overlap area. For example, for an overlap area 240, 240a, 240b that is rectangular and has a uniform width, as disclosed in Figures 1c, 2a, and 2b, the image overlap size may be set to the width of the overlap.

[0048] Further, a candidate area overlap ratio is determined S320 indicating the overlap ratio between the first candidate area 260a, 260b and the second candidate area 270a, 270b. Any type of normalization related to the size of the first candidate area 260a, 260b and the second candidate area 270a, 270b, typically related to the size of the union of the first candidate area 260a, 260b and the second candidate area 270a, 270b, may be used. For example, the candidate area overlap ratio may be determined as the intersection of the first candidate area 260a, 260b and the second candidate area 270a, 270b divided by the union of the first candidate area 260a, 260b and the second candidate area 270a, 270b.

[0049] Next, an integrating threshold is determined S330 based on the size of the image overlap. The integrating threshold may be determined to increase with the size of the image overlap. For example, the integrating threshold may be determined to be proportional to the size of the image overlap. Thus, for a given overlap size, the integrating threshold is determined to be that given overlap size multiplied with a proportionality constant.

[0050] The first candidate area 260a, 260b and the second candidate area 270a, 270b are merged S340 to form a merged candidate area, with the candidate area overlap ratio being greater than a merger threshold C335. The merged candidate area may consist of the union of the first candidate area 260a, 260b and the second candidate area 270a, 270b.

[0051] By determining the merging threshold based on the size of image overlap, the risk of not merging a first candidate area and a second candidate area even though they are associated with the same feature is reduced, thus reducing the risk that the same feature in separate images will be identified as two separate features.

[0052] The method 300 may be well adapted to accommodate different shapes and orientations of features resulting from the identification of different shapes and orientations of the candidate areas. For example, if the feature is elongated, the different orientations of the feature will result in a different proportion of the feature, and therefore a different proportion of the candidate area identified will typically be in the overlap region. For example, with reference to Figures 2a, 2b, and 2c, if the features 250, 250a, 250c are horizontal instead of vertical, a smaller proportion of the feature will be in the overlap region 240, 240a, 240b. Thus, to account for this, the size of the intersection of the overlap region 240, 240a, 240b and the junction of the first candidate area 260a, 260b and the second candidate area 270a, 270b may be determined. The merging threshold may be determined based on the size of the image overlap, with the merging threshold being based on the size of the intersection of the overlapping regions 240, 240a, 240b and the union of the first candidate area 260a, 260b and the second candidate area 270a, 270b. For example, the merging threshold may be determined to be proportional to the size of the intersection of the overlapping regions 240, 240a, 240b and the union of the first candidate area 260a, 260b and the second candidate area 270a, 270b. Thus, for a given size of the intersection of the overlapping regions 240, 240a, 240b and the union of the first candidate area 260a, 260b and the second candidate area 270a, 270b, the integration threshold is determined to be the given size of the intersection of the overlapping regions 240, 240a, 240b and the union of the first candidate area 260a, 260b and the second candidate area 270a, 270b multiplied with a proportionality constant. Such a match is also suitable when the overlapping regions do not have a uniform width.

[0053] Although embodiments of method 300 have been described in relation to two images, embodiments of the method are also applicable when there are three or more images, in which case embodiments of method 300 may be applied to the three or more images in one step for all of the images, or recursively, i.e., first to the first two images that have overlaps, followed by further images of the three or more images that have overlaps with either of the two first images, resulting from the first two images, and so on.

[0054] FIG. 4 shows a schematic diagram related to an embodiment of a device 400 of the present disclosure for merging a first candidate area associated with a candidate feature in a first image and a second candidate area associated with a candidate feature in a second image, where the first image and the second image have an overlapping region and at least a portion of the first candidate area and at least a portion of the second candidate area are located in the overlapping region. The device 400 may be, for example, a camera. The device 400 includes a circuit 410. The circuit 410 is configured to perform functions 432, 434, 436, 438 of the device 400. The circuit 410 may include a processor 412, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 412 is configured to execute program code. The program code may be, for example, configured to perform functions 432, 434, 436, 438 of the device 400.

[0055] The device 400 may further include a memory 430. The memory 430 may be one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. In a typical configuration, the memory 430 may include a non-volatile memory for long-term storage of data and a volatile memory that serves as a system memory for the circuit 410. The memory 430 may exchange data with the circuit 410 via a data bus. There may also be associated control lines and an address bus between the memory 430 and the circuit 410.

[0056] The functions 432, 434, 436, 438 of the device 400 may be embodied in the form of executable logic routines (e.g. lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (memory) 430 of the device 400 and may be executed by the circuitry 410, for example using a processor 412 therein. Furthermore, the functions 432, 434, 436, 438 of the device 400 may be stand-alone software applications or may form part of a software application. The described functions may be considered as methods that a processing unit, for example the processor 412 of the circuitry 410, is configured to perform. Also, while the described functions 432, 434, 436, 438 may be implemented in software, such functions may likewise be implemented via dedicated hardware or firmware, or some combination of hardware, firmware, and / or software.

[0057] The circuit 410 is configured to execute a determine image overlap size function 432 configured to determine an image overlap size indicating the size of an overlapping area between a first image and a second image.

[0058] The circuit 410 is further configured to perform a candidate area overlap ratio determine function 434 configured to determine a candidate area overlap ratio indicative of an overlap between the first candidate area and the second candidate area.

[0059] The circuit 410 is further configured to perform a determine merging threshold function 436 configured to determine the merging threshold based on the size of the image overlap.

[0060] The circuit 410 is further configured to perform a merging function 438 configured to merge the first candidate area and the second candidate area, if an overlap ratio of the candidate areas is greater than a merging threshold, thereby forming a merged candidate area.

[0061] The device 400 and the functions 432, 434, 436, 438 performed by the circuit 410 may further be adapted as the method 300 and corresponding steps of the method 300, respectively, described in connection with FIG.

[0062] Those skilled in the art will understand that the present invention is not limited to the above-described embodiments. On the contrary, many modifications and variations are possible within the scope of the claims. Such modifications and variations can be understood and effected by those skilled in the art in the practice of the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

Claims

1. 1. A computer-implemented method for merging a first candidate area associated with a candidate feature in a first image and a second candidate area associated with a candidate feature in a second image, the first image and the second image having an overlapping region, and at least a portion of the first candidate area and at least a portion of the second candidate area being located in the overlapping region; determining an image overlap size indicating a size of the overlapping area between the first image and the second image; determining a candidate area overlap ratio indicative of an overlap ratio between the first candidate area and the second candidate area; determining an integration threshold based on a size of the image overlap; merging the first candidate area and the second candidate area, where an overlap ratio of the candidate areas is greater than the merging threshold, thereby forming a merged candidate area; The method includes:

2. The method of claim 1 , wherein the merging threshold is determined to be proportional to a size of the image overlap.

3. 2. The method of claim 1, wherein the candidate area overlap ratio is determined as an intersection of the first candidate area and the second candidate area divided by a union of the first candidate area and the second candidate area.

4. The method of claim 1 , wherein the first candidate area and the second candidate area are identified in coordinates of a combined image that includes the first image and the second image.

5. 2. The method of claim 1, wherein the first image is a first transformed view of a first portion of a fisheye image and the second image is a second transformed view of a second portion of the fisheye image, and the first candidate area and the second candidate area are identified in coordinates of the fisheye image.

6. The method of claim 1 , wherein the combined candidate area comprises a union of the first candidate area and the second candidate area.

7. The method of claim 1 , wherein the first and second candidate areas are one of a bounding box, a pixel mask, and a polygon area.

8. A non-transitory computer readable storage medium having stored thereon instructions which, when executed on a device having processing capabilities, perform the method of any one of claims 1 to 7.

9. 1. A device for merging a first candidate area associated with a candidate feature in a first image and a second candidate area associated with a candidate feature in a second image, the first image and the second image having an overlapping region, and at least a portion of the first candidate area and at least a portion of the second candidate area being located in the overlapping region; The device comprises: an image overlap size determiner configured to determine an image overlap size indicating a size of the overlapping area between the first image and the second image; a candidate area overlap ratio determiner configured to determine a candidate area overlap ratio indicative of an overlap between the first candidate area and the second candidate area; a merger threshold determining function configured to determine a merger threshold based on a size of the overlap of the images; a merging function configured to merge the first candidate area and the second candidate area, if an overlap ratio of the candidate areas is greater than the merging threshold, thereby forming a merged candidate area; A device comprising circuitry configured to perform the steps of:

10. The device of claim 9 , wherein the function of determining the merging threshold is configured to determine the merging threshold to be proportional to a size of the overlap of the images.

11. 10. The device of claim 9, wherein the function for determining an overlap ratio of the candidate areas is configured to determine the overlap ratio of the candidate areas as an intersection of the first candidate area and the second candidate area divided by a union of the first candidate area and the second candidate area.

12. The device of claim 9 , wherein the first candidate area and the second candidate area are identified in coordinates of a combined image including the first image and the second image.

13. 10. The device of claim 9, wherein the first image is a first transformed view of a first portion of a fisheye image and the second image is a second transformed view of a second portion of the fisheye image, and the first candidate area and the second candidate area are identified in coordinates of the fisheye image.

14. The device of claim 9 , wherein the combined candidate area comprises a union of the first candidate area and the second candidate area.

15. The device of claim 9 , wherein the first and second candidate areas are one of a bounding box, a pixel mask, and a polygonal area.

Images