Method and image processing device for assessing alignment of stitched images - Patents.com

Abstract

To provide a method, a program, and a carrier for dynamically calibrating stitching of images fetched by a plurality of image sensors.SOLUTION: A method aligns a first image frame of a first image from a first image sensor and a first stream of a second image from a second image sensor according to first alignment, and blends them to obtain a first stitched image frame. The method obtains a second stitched image frame in accordance with a second blending function, and evaluates the first alignment on the basis of, a difference pattern calculated on the basis of values of pixels of the first stitched image frame and the second stitched image frame.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] Embodiments herein relate to a method and image processing apparatus for assessing the alignment of stitched images. A corresponding computer program and computer program carrier are also disclosed. [Background technology]

[0002] Public surveillance using imaging, particularly video imaging, is common in many areas around the world. Areas that may require monitoring are, for example, banks, stores, and other areas where security is required, such as schools and government facilities. Other areas that may require monitoring are processing, manufacturing, and logistics applications, where video surveillance is primarily used to monitor processes.

[0003] For some applications, a wide field of view of the video is required, at least occasionally. This can be achieved by several means. For example, a single-sensor camera can be equipped with pan-tilt-zoom technology, which moves the camera to change the field of view. This solution is usually quite expensive compared to fixed cameras. Single-sensor fisheye cameras offer an affordable panoramic view, but are limited in their ability to capture more detail at greater distances. Another solution is to use a multi-sensor panoramic camera, where the panoramic image is generated by stitching the images of the single sensors.

[0004] Image stitching or photo stitching is the process of combining multiple captured images with overlapping fields of view to produce one uniform image, e.g., a segmented high-resolution image, or in some cases a panoramic image. For example, image data from overlapping image sensors may be blended with different weights where there is overlap between images from the sensors. Most approaches to image stitching, commonly performed in software, require near-precise overlap and identical exposure between images to produce a seamless result. It is important that the images from the sensors are correctly aligned in the overlapping areas. If not, errors may appear, such as parts of the same object appearing twice in the overlap or parts of the object being missing.

[0005] Generally, alignment errors can be categorized as, for example, translational alignment errors, rotational alignment errors, and alignment errors due to skew. Other alignment errors also exist.

[0006] Alignment between images can be found from knowing the mounting location of the sensor or from analyzing the separate images to be stitched to find common features and use these to align and / or warp the images. One way to align the original images is to create tables that map coordinates in the original images to new coordinates in the final image. These tables may depend on certain parameters of the camera, such as parameters for the mounting location of the sensor. The parameters may be pre-installed in the camera after factory calibration and later supplied by the camera. Even though factory calibration usually introduces only minor alignment errors, some adjustments may still be required to the alignment of the original images when installing the camera in the field. For example, alignment adjustments may be required if the camera is mechanically disturbed during transportation, or if the field environment is significantly different from the factory test environment, or if the objects captured in the field are not at a similar distance to the objects used to calibrate the factory alignment.

[0007] Even if recalibration is performed when the camera is installed in the field, such recalibration must be determined offline, i.e., not a runtime / live view of the camera, which means that the image stream from the camera will be unavailable for a period of time.

[0008] A problem with alignment methods that are performed during camera runtime is that they may require a separate image stream for the evaluation of the alignment, e.g., unaligned images may be evaluated. Such alignment methods require extra complexity and processing power, and potentially extra bandwidth on the network over which such data is transmitted, compared to the case where no alignment was performed.

[0009] Other alignment methods performed during camera runtime can affect the quality of the stitched image, which can negatively impact any video processing that occurs after stitching and ultimately be perceived as a lower quality image by the final observer.

[0010] Prior art US Pat. No. 5,399,433 discloses a method for stitching images that includes evaluating stitching quality. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] US Patent No. 8,600,193 B2 Specification Summary of the Invention [Problem to be solved by the invention]

[0012] Therefore, an objective of the embodiments herein may be to eliminate some of the above-mentioned problems, or at least reduce their effects. In particular, the objective may be to dynamically improve the alignment of stitched images, in other words to dynamically calibrate the stitching of images captured by multiple image sensors at run-time of an image processing device performing the stitching operation. The image processing device may be a camera. Performing the calibration at run-time of the image processing device may mean performing the calibration while the image processing device is in use.

[0013] As part of improving the alignment or calibration, an evaluation of the alignment may be performed, which may include determining the degree or quality of the alignment, such as determining the magnitude, type, and / or direction of the alignment or misalignment error. The terms degree of alignment, quality of alignment, and alignment quality are used interchangeably herein and have the same meaning. Similarly, the terms alignment error and misalignment error are used interchangeably herein.

[0014] According to one aspect, this object is achieved by a method performed by an image processing device, such as an image capture device. The method may be for evaluating alignment of stitched images from two image sensors having overlapping fields of view, for example for evaluating alignment of a first image stream from a first image sensor and an alignment of a second image stream from a second image sensor, the first and second image sensors having overlapping fields of view. The image sensors may be comprised within the image capture device.

[0015] The method includes aligning a first image frame of a first image stream from a first image sensor with a further first image frame of a second image stream from a second image sensor in accordance with a first alignment, and aligning a second image frame of the first image stream with a further second image frame of the second image stream in accordance with the first alignment.

[0016] The method further includes obtaining a first stitched image frame by blending the aligned first image frame and the further first image frame in a first blending region of the first and further first image frames according to a first blending function, and obtaining a second stitched image frame by blending the aligned second image frame and the further second image frame in a second blending region of the second and further second image frames according to a second blending function different from the first blending function.

[0017] The method further includes evaluating the first alignment based on a difference pattern calculated based on pixel values ​​of the first stitched frame and the second stitched frame, the pixels being included within at least one static portion of the first and second blend regions in the respective first and second stitched image frames.

[0018] According to another aspect, the above object is achieved by an image processing device arranged to carry out the above method.

[0019] According to a further aspect, the above object is achieved by a computer program and a computer program carrier corresponding to the above aspect.

[0020] Since the difference pattern is based on at least one static part, the difference pattern does not depend on the motion in the scene between different images taken after each other. Instead, the difference pattern depends on the change in the blending function and the alignment. The change in the blending function has the least effect on the difference pattern of the most aligned image. For example, for a blending function where the sum of the blending functions of the first image and the further first image is equal to the sum of the blending functions of the second image and the further second image, a different blending function should result in a zero difference pattern for perfectly aligned images. For images with misalignment errors, a different blending function will produce a non-zero difference pattern.

[0021] Thus, since the first stitched frame and the second stitched frame are stitched using different blend functions, and the difference pattern is based on at least one static portion, it is possible to dynamically evaluate and improve the alignment of the stitched images by evaluating the first alignment based on a difference pattern calculated based on pixel values ​​of the first stitched image frame and the second stitched image frame.

[0022] A further advantage of embodiments herein is that the alignment can be evaluated during runtime of the imaging device, such that dynamic recalibration of the alignment can be performed without stopping operation of the imaging device, in particular, a sufficiently good evaluation of the alignment will have little impact on the resulting video stream and therefore will not be noticed by the final observer.

[0023] Furthermore, a separate image stream is not required to assess alignment since the difference pattern can be calculated from the stitched image stream.

[0024] Furthermore, a further advantage of the embodiments herein is that they can simplify the evaluation of complex non-linear misalignment errors in stitched images, such as rotation of the original stitched images, by evaluating the misalignment as a translational misalignment in many small static parts spread across the blend region.

[0025] In the figures, features that appear in some embodiments are indicated by dashed lines.

[0026] The various aspects of the embodiments disclosed herein, including particular features and advantages thereof, will be readily understood from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0027] [Figure 1] 1 is a schematic diagram illustrating an exemplary embodiment of an image processing device, such as an image capture device, herein; [Diagram 2] 1 illustrates an exemplary embodiment of a video network system including an image processing device. [Figure 3a] FIG. 1 is a schematic diagram illustrating an exemplary embodiment herein. [Figure 3b] FIG. 1 is a schematic diagram illustrating an exemplary embodiment herein. [Figure 3c] FIG. 1 is a schematic diagram illustrating an exemplary embodiment herein. [Figure 3d] 1 illustrates an exemplary embodiment of stitching two image frames. [Figure 3e] 4 illustrates an example embodiment of a blending function. [Figure 3f] 4 illustrates an example embodiment of multiple blending functions. [Figure 4] 1 is a flow chart illustrating an embodiment of a method in an image processing apparatus. [Diagram 5] The difference measure as a function of alignment is shown. [Figure 6] 1 is a block diagram showing an embodiment of an image processing apparatus; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] As mentioned above, it would be interesting to improve the alignment of stitched images from two or more sensors. As an example, the alignment can be determined at the factory, for example, from knowledge of the mounting positions of the sensors.

[0029] However, when installing the camera in the field, the alignment of the original images may still require some adjustment.

[0030] As mentioned above, a problem with alignment methods that are performed during the runtime of the camera is that they may require one or more separate image streams to evaluate the alignment.

[0031] In embodiments herein, these problems are solved by a method for evaluating the alignment of two image streams from sensors with overlapping fields of view. More specifically, these problems can be solved by dynamically improving the alignment of the stitched images by embedding information in the stitched image stream, i.e., the image stream of the stitched images, that can be used to analyze the alignment.

[0032] The embedded information is obtained by calculating a difference pattern, in other words a difference image, between two stitched images formed by blending two images emanating from an image sensor respectively by varying a blending function of the stitched original images and then evaluating the alignment, e.g., by determining the degree or quality of the alignment. For example, the difference pattern may be calculated by calculating a difference based on corresponding pixel values ​​of the two stitched images. In some embodiments herein, the difference pattern is calculated by calculating a difference between weighted pixel values ​​of the two stitched images. In some embodiments, the weights are equal to 1 such that the difference pattern is calculated by calculating a difference between pixel values ​​of the two stitched images.

[0033] In some embodiments herein, evaluating the alignment based on the difference pattern includes comparing the difference pattern for the first alignment to difference patterns for one or more further alignments, such as the second alignment. The alignment that results in the smallest difference pattern can then be selected. For these embodiments, the difference pattern may be calculated by calculating the difference between pixel values ​​of the two stitched images.

[0034] In some other embodiments, a difference pattern, such as a single difference pattern, is used as an input to the neural network along with one of the stitched images used to generate the difference pattern. After initial training, the neural network can evaluate the alignment based on the difference pattern and the stitched image.

[0035] In yet some other embodiments, the difference pattern is used (together with one of the stitched images) to calculate a reconstruction of the original image in the stationary portion. The assessment of the alignment can then be based on any known method for extracting differences and / or similarities between these original images and assessing these differences and / or similarities. Such feature matching methods can include sum of squared differences (SSD), normalized cross-correlation (NCC), or scale-invariant feature transform (SIFT).

[0036] In other words, embodiments herein improve the alignment of stitched image frames by varying the blending functions, i.e., blending coefficients or blending weights, of the stitched image frames at run-time in an image processing device such as a camera, and considering the effect of this variation on the difference patterns in static portions, such as a small patch of 8x8 pixels of the blending region contained in the overlap region between the original images. The static portions are portions that remain invariant between different image frames when the blending functions are the same.

[0037] If the original images are well aligned, the blending variations should have minimal impact on the stitched image, but if the original images are not well aligned, the blending variations will cause pixel value variations in the static parts of the overlapped image regions, i.e., the blending variations will cause different stitched images blended with different blending functions to be different, i.e., constitute different pixel values ​​at corresponding pixel locations. The difference based on the pixel values ​​at corresponding pixel locations can be monitored and evaluated, and based on the evaluation of the difference, a determination can be made as to whether the alignment is good enough.

[0038] An advantage of the embodiments herein is that the changes are not so large as to be very noticeable to an observer of the stitched image. Furthermore, the embodiments herein allow for the simplification of the evaluation of complex non-linear misalignment errors in the stitched images, such as rotation of the original images being stitched, by evaluating the misalignment as a translational misalignment in many small static parts spread over the blend region.

[0039] Embodiments herein may be implemented in one or more image processing devices, such as an image capture device, e.g., a digital camera. Figure 1 illustrates various exemplary image capture devices 110. Image capture device 110 may be, for example, a video camera 120, such as a camcorder, video recorder, surveillance camera, or monitoring camera, a digital camera, a smartphone 130 including an image sensor, or a vehicle 140 including an image sensor.

[0040] 2 illustrates an example video network system 250 in which embodiments herein may be implemented. The video network system 250 may include video cameras 120, 220 that may capture digital images 201, such as digital video images, and perform image processing. A video server 260 in FIG. 2 may obtain images from the video cameras 120, 220, for example, over a network or the like.

[0041] A video server is a computer-based device dedicated to video distribution. Video servers are used in many applications and often have additional features and capabilities that address the needs of a particular application. For example, video servers used in security, surveillance and inspection applications are typically designed to capture video from one or more cameras and distribute the video over a computer network. In video production and broadcast applications, a video server may have the capability to record and play back recorded video and distribute many video streams simultaneously. In FIG. 2, a video server 260 is connected to image capture devices exemplified by video cameras 120, 220 via a network 250. The video server 260 may further be connected to a video storage device 270 for storing video images and / or to a monitor 280 for displaying the video images.

[0042] The embodiments herein may be implemented in multiple image processing devices. For example, in some embodiments herein, the method for evaluating the alignment of two image streams is performed in a distributed manner in both the image capture devices, such as the video cameras 120, 220, and in one or more further image processing devices, such as the video server 250. The one or more further image processing devices may also be configured in a video management system (VMS), or a computer cloud. More specifically, parts of the method, such as image alignment and blending, may be performed by the image capture devices, and other parts, such as evaluation of alignment, may be performed by one or more further image processing devices. This is possible because information related to blending is embedded in the image streams output from the image capture devices.

[0043] To better understand the embodiments herein, an imaging system will first be described.

[0044] FIG. 3a is a schematic diagram of an imaging system 300, in this case a digital video camera such as the video cameras 120, 220. The imaging system images a scene on one or more image sensors, for example on a first image sensor 301a and a second image sensor 301b. The first and second image sensors 301a, 301b may be equipped with respective Bayer filters such that different pixels receive radiation in specific wavelength ranges in a known pattern. Typically, each pixel of a captured image is represented by one or more values ​​that represent the intensity of the captured light in a specific wavelength band. These values ​​are usually referred to as color components or color channels. The term "image" may refer to an image frame or a video frame that includes information derived from the image sensors that captured the image.

[0045] After reading the signals of the individual sensor pixels of the image sensors 301a, 301b, different image processing operations can be performed by the image processing pipeline 302. Generally, for video processing, an image is composed of image streams. Figure 3a shows a first image stream 310a from a first image sensor 301a and a second image stream 310b from a second image sensor 301b.

[0046] Image processing may include demosaicing, color correction, noise filtering (to remove spatial and / or temporal noise), distortion correction (e.g., to remove barrel distortion effects), global and / or local tone mapping (e.g., to enable imaging of scenes containing a wide range of intensities), transformations (e.g., scale and rotate), flat-field correction (e.g., to remove vignetting effects), application of overlays (e.g., privacy masks, legends), etc. The image processing pipeline 302 may also be associated with an analytics engine that performs object detection, recognition, alerts, etc.

[0047] The image processing pipeline 302 may consist of an image processor 302a and a video post processor 302b. The image processor 302a may, for example, perform image stabilization, apply noise filtering, distortion correction, global and / or local tone mapping, transformations, and flat-field correction. The video post processor 302b may, for example, crop portions of an image, apply overlays, and include an analysis engine.

[0048] Specifically, in embodiments herein, processing of the images also includes stitching together two images from the first and second image sensors 301a, 301b. The stitching may be performed relatively late in the image processing pipeline 302, for example in the video post-processing section 302b. In some embodiments, at least noise filtering is performed before stitching is performed. Figure 3a illustrates generally how the first and second image streams 310a, 310b are combined to form a stitched image stream 310c.

[0049] Fig. 3b shows a schematic representation of such an image stitching process, in which a first image frame 311a and a further first image frame 311b from the first and second image sensors 301a, 301b, respectively, are stitched to generate a first stitched image frame 321. In a second such stitching process, a second image frame 312a and a further second image frame 312b from the first and second image sensors 301a, 301b, respectively, are stitched to generate a second stitched image frame 322. The first image frame 311a and the further first image frame 311b may have been captured at the same or a similar time in order to minimize variations between the two image frames 311a, 311b due to object movements in the scene captured by the image frames. Similarly, the second image frame 312a and the further second image frame 312b may have been captured at the same or a similar time.

[0050] It is also possible to stitch together more than two frames.

[0051] It should be noted that in the case of a video camera, the output frame rate from the video camera, e.g., a rate corresponding to the display rate of image frames on a display, such as 30 fps or 60 fps, can place a limit on the time that can be spent on image processing, especially when a live video stream is required. The embodiments herein can be performed without reducing the frame rate. In particular, changes in the blending function can be made without affecting the frame rate of the stitched images.

[0052] Following the image processing pipeline 302, the image may be forwarded to an encoder 303 where the information in the image frames is encoded according to an encoding protocol such as H.264 and forwarded to a receiving client, exemplified herein as a monitor 280, a video server 260, a storage device 270, etc.

[0053] An exemplary scenario and an exemplary embodiment of the present specification will now be described with reference to Fig. 3c. Fig. 3c illustrates an exemplary image processing device in the form of a scene including a video camera 325 and a plurality of objects. The video camera 325 may include the imaging system 300 of Fig. 3a described above. For example, the video camera 325 may comprise a first image sensor 301a and a second image sensor 301b. The first and second image sensors have an overlapping field of view 328, shown as a hatched area in Fig. 3c. A static object 330 is located within the overlapping field of view 328, such that at least a portion of the static object 330 is imaged on both the first image sensor 301a and the second image sensor 301b.

[0054] Next, with reference to Figures 3d, 3e and 3f and the flow chart of Figure 4, and further with reference to Figures 3a-3c, an exemplary method according to an embodiment herein will be described.

[0055] In order to stitch two image frames together, the image frames are aligned. The initial alignment may be performed at the factory. However, when the imaging device is installed and operational in the field, realignment may be required from time to time. Therefore, in the following, a method for such realignment, i.e., alignment recalibration, is presented. Alignment recalibration involves an evaluation of the current alignment or misalignment, which may include determining the degree of alignment and / or calculating the misalignment error.

[0056] Next, embodiments of a method for assessing the alignment of two image streams from sensors with overlapping fields of view are described. First, a brief description of some specific embodiments is provided to explain how the various operations interact.

[0057] The method may comprise the following acts.

[0058] 1. Prepare a first stitched image frame 321 by arranging the first and further first image frames 311a, 311b according to a first alignment and blending the aligned first and further first image frames 311a, 311b in a first blending region 331a. The first blending region 331a where the aligned first and further first image frames 311a, 311b are blended is composed of a first overlap region 332a of the aligned first and further first image frames 311a, 311b. The first overlap region 332a may be defined as a region of each of the first and further first image frames 311a, 311b that is composed of pixels having corresponding content of the scene.

[0059] 2. Select a number of static portions 334a, 334b ​​of the first blending region 331a in the stitched image frames 321, 322, and for each such static portion 334a, 334b, determine the alignment quality as follows.

[0060] For the second stitched image frame 322, increase the blending weight of one of the second and further second images 321a, 312b in the second blending region 331b by a fixed amount, e.g., 10%, and decrease the weight of the other image frame by a corresponding amount. The second blending region 331b, where the aligned first and further first image frames 311a, 311b are blended, is comprised of the second overlap region 332b of the aligned second and further second image frames 311a, 311b.

[0061] b. Calculate a difference pattern based on pixel values ​​of the second stitched image frame 322 and the first stitched image frame 321 in the static portions 334a, 334b ​​of the blending regions 331a, 331b. For example, calculate a difference pattern between the second stitched image frame 322 and the first stitched image frame 321 in the static portions 334a, 334b ​​of the blending regions 331a, 331b.

[0062] 3. Evaluate the difference patterns to determine hypotheses regarding improved alignments, which in a first version consists of:

[0063] If the difference pattern is greater than zero, the alignment is not perfect and choose a different alignment and test again.

[0064] b. If the difference pattern is equal to zero, the alignment is successful and the evaluation is complete.

[0065] c. The assessment, in its second version, is constructed as follows:

[0066] i. Use a pre-trained neural network to determine an improved alignment based on the difference patterns in the blend regions 331a, 331b and the stitched images 321, 322.

[0067] Each operation is described in more detail below. The method herein for evaluating the alignment of stitched images from two image sensors with overlapping fields of view, in other words for evaluating the alignment of two image streams from two image sensors with overlapping fields of view, for example for evaluating the alignment of a first image stream 310a from a first image sensor 301a and a second image stream 310b from a second image sensor 301b, may be implemented in an image processing device such as any of the image capture devices 110 of FIG. 2a, in particular the video camera 325 of FIG. 3c. The image sensors 301a, 301b may be included within the image capture device 110, such as the video camera 325.

[0068] As mentioned above, even if a factory calibration is performed, the alignment of the original images may still require some adjustment when the video camera 325 is installed in the field. Implementation of the embodiments herein in the video camera 325 is particularly relevant when performed at the runtime of the video camera 325, as it can be performed in the video camera 325 without providing a separate image stream for alignment evaluation and with only minor impact on the stitched image stream. Thus, the method can be performed without significant impact on the normal use of the image processing device.

[0069] 4 may be performed in the following example order: In other examples, the order may differ from that described below.

[0070] Operation 401a 3d shows a schematic overlay of two image frames emitted from two image sensors according to some alignments with some alignment errors. The two image frames are, for example, a first image frame 311a and a further first image frame 311b. In another example, the two image frames are a second image frame 312a and a further second image frame 312b.

[0071] As mentioned above, the calculation or determination of the alignment used to stitch the first image frame 311a and the further first image frame 311b may have been performed, for example at a factory, before the method of evaluating the alignment begins. For example, it is possible to use a table that maps coordinates in the original images 311a, 311b to their new coordinates in the stitched image 321. Other embodiments use other tables that map coordinates in the stitched image 321 to coordinates in the original images 311a, 311b. These tables may depend on the specific parameters of the video camera 325, such as parameters related to the mounting positions of the sensors 301a, 301b. Thus, operation 401a and the next operation 401b align the two image frames according to a known or determined alignment, without evaluating and updating the alignment. In other words, operations 401a and 401b combine or overlay the two image frames according to a known or determined alignment. This is part of the normal stitching process. The evaluation of the alignment is performed in operation 406 below.

[0072] In Fig. 3d, the alignment error is illustrated with a translational misalignment in the vertical pixel direction. Other misalignments, such as rotational and / or diagonal misalignments, may also occur, and the following description of the embodiments herein is equally applicable to such other misalignment errors. More specifically, Fig. 3d illustrates an overlay or alignment of a first image frame 311a of a first image stream 310a from a first image sensor 301a with a further first image frame 311b of a second image stream 310b from a second image sensor 301b according to a first alignment. The first alignment is associated with a first alignment error. The first image frame 311a and the further first image frame 311b may have been captured at the same or similar time such that variations between the two image frames 311a, 311b, e.g. due to the movement of objects in the scene, are as small as possible within the first blending region 331a. It is desirable to have as little variation as possible due to the movement of objects in the scene, since the embodiments herein exploit, or in other words analyze, such variations in the stitched images caused by differences in the blending process.

[0073] Therefore, to stitch the first image frame 311a and the further first image frame 311b, they are aligned according to the first alignment.

[0074] Operation 401b In a manner corresponding to operation 401a above, the image processing device aligns a second image frame 312a of the first image stream 310a from the first image sensor 301a and a further second image frame 312b of the second image stream 310b from the second image sensor 301b in accordance with the first alignment.

[0075] The second image frame 312a and the further second image frame 312b may have been captured at the same or a similar point in time such that variations between the two image frames 312a, 312b due to, for example, object movement in the scene are as small as possible within the second blending region 331b. However, relative to the first and further first image frames 311a, 311b, the second image frame 312a and the further second image frame 312b may have been captured at a different point in time, for example a later point in time.

[0076] Operation 402 When the first image frame 311a and the further first image frame 311b are aligned or combined, for example according to operations 401a and 401b above, the image processing device obtains a first stitched image frame 321 by blending the aligned first image frame 311a and the further first image frame 311b within a first blending region 331a of the first and further first image frames 311a, 311b according to a first blending function 333a, an example of which is shown in FIG. 3e.

[0077] In embodiments herein, the blending function may define respective blending weights or blending coefficients of an image frame to be blended with a further corresponding image frame as a function of pixel location within a blending region of the image. Thus, the blending function may be composed of a plurality of respective functions defining respective blending weights of two corresponding image frames. In particular, the first blending function 333a may define respective blending weights of the aligned first image frame 311a and the further first image frame 311b as a function of pixel location within the first blending region 331a.

[0078] The blending function may be any known blending function, for example an alpha blending function.

[0079] There are several versions of alpha blending, but the common concept is that each pixel is weighted with a blend function, for example a linear blend function. Each pixel is then prioritized according to where it is in each respective blend region 331a, 331b, or where it is in each respective static portion 334a, 334b. An example implementation uses a mask and multiplies each static portion 334a, 334b ​​with either the mask or the 1-mask.

[0080] Alpha blending is used to make the transition between the two images smoother. However, this also means that edges and artifacts present in inaccurate stitching are also smoothed. This can make them difficult to detect, and even their magnitudes difficult to detect. However, in the present embodiment, it is possible to detect these errors even with alpha blending, since the alignment is evaluated based on a difference pattern calculated based on pixel values ​​of the two stitched images due to the variation of the blending function.

[0081] The advantage of using an alpha blending function to evaluate alignment based on the difference pattern is that the difference pattern is calculated with the same function for each pixel, which means that the total difference value for the static portion 334 is based on the sum of the differences for each pixel. An example is given below. In this example, it is assumed that the sum of the blending weights for a particular pixel for the first stitched image frame 321 is the same as the sum of the blending weights for the same pixel for the second stitched image frame 322. The sum can correspond to, for example, 1 or 100%.

[0082] In Fig. 3e, the first blending function 333a is a function that varies linearly within the first blending region 331a shown below the first blending function 333a in Fig. 3e. In this example, the linearly varying function includes a linear variation of the blending weights of each of the first and further first images in percentages as a function of horizontal pixel position within the first blending region 331a. The first blending weights shown in Fig. 3e with hatched lines may be applied to the first image frame 311a, and the further first blending weights shown in Fig. 3e with solid lines may be applied to the further first image frame 311b.

[0083] According to Fig. 3e, for pixel positions in the first image frame 311a that do not overlap with the further first image frame 311b, i.e. pixel positions to the right of the blending area 331a in Fig. 3e, the first blending weight is 100%. For pixel positions within the blending area 331a, the first blending weight varies from 100% at the pixel position where the further first image frame 311b ends to 0% at the pixel position where the first image frame 311a ends. And for pixel positions in the further first image frame 311b that do not overlap with the first image frame 311a, the first blending weight is 0%.

[0084] Further first blending weights for the further first image frame 311b may be defined correspondingly as above.

[0085] Here, for a particular pixel location of the static portion 334 of the first stitched image frame 321, the first blending weight applied to the first image frame 311a may be 30%. Then, for the same pixel location, the further first blending weight applied to the further first image frame 311b may be 70%. For the corresponding second stitched image frame 322 and second blending function, the second blending weight applied to the second image frame 312a may be 40%. Then, for the same pixel location, the further second blending weight applied to the further second image frame 312b may be 60%.

[0086] The first blending function 333a may also be described as being composed of a number of blending functions, for example respective blending functions corresponding to the first image frame and further first image frames.

[0087] operation 403 In a corresponding manner similar to operation 401a above, the image processing device 110 blends the aligned second image frame 312a and the further second image frame 312b in a second blending region 331b of the second and further second image frames 312a, 312b according to a second blending function 333b different from the first blending function 333a to obtain a second stitched image frame 322. Figure 3f shows examples of both the first blending function 333a shown by a solid line combined with a hatched line and the second blending function 333b shown by a dotted line combined with a dotted hatched line as a function of pixel position.

[0088] Similar to the first blending function 333a, the second blending function 333b can define respective blending weights of the aligned second image frame 312a and the further second image frame 312b as a function of pixel position within the second blending region 331b.

[0089] In embodiments herein, the first overlap region 332a may cover the same pixels as the second overlap region 332b. The first and second overlap regions 332a, 332b are shown in FIG. 3f as unbroken horizontal lines below the pixel position axis. Even so, the first blend region 331a and the second blend region 331b may be different. The second blend region 331b may be, for example, translated relative to the first blend region 331a. In some embodiments herein, each blend region 331a, 331b is defined as an overlap region that overlaps a linear change in blend weights of the blend function.

[0090] In FIG. 3f, the different regions are shown as different types of horizontal lines positioned below the horizontal pixel position axis.

[0091] The first blending region 331a is illustrated as a thick horizontal line with a thin horizontal line below the thick horizontal line, and the second blending region 331b is illustrated as a thick horizontal line with a thin horizontal line above the thick horizontal line. However, it is also possible for the second blending region 331b to cover the same area as the first blending region 331a. For example, the second blending region 331b and the first blending region 331a may be defined to cover the entire overlapping region 332a, 332b illustrated by the two thick solid lines.

[0092] In any case, a region covering linearly varying portions of the first and second blend regions 331a, 331b, illustrated by two thick hatched horizontal lines, may be used to assess the alignment, e.g., calculate the difference pattern.

[0093] A second blending weight, shown by dotted lines in FIG. 3f, may be applied to the second image frame 311b, and a further second blending weight, shown by dotted lines in FIG. 3f, may be applied to the further second image frame 312b.

[0094] As shown in Fig. 3f, the second blending function 333b may differ from the first blending function in that the second blending weight applied to the second image frame 312a is higher than the first blending weight applied to the first image frame 311a in the blending region 331a, while the further second blending weight applied to the further second image frame 312b is lower than the further first blending weight applied to the further first image frame 311b in the blending region 331a. This can be obtained by translating parts of the first and second blending functions 333a, 333b that vary linearly with respect to each other. This means that the overlapping region 332a, 332b may be larger than the first and second blending regions 331a, 331b used for blending. The static portions 334a, 334b ​​may be determined to be included in the part of the stitched images 321, 322 that consists only of linearly varying blending functions. That is, the part of the overlapping region that consists of static blending functions may be discarded. For example, the second blending weight may vary between 10% and 100% within the second blending region 331b, while the first blending weight may vary between 0% and 100% within the corresponding first blending region 331a. Similarly, a further second blending weight may vary between 90% and 0% within the second blending region 331b, while the first blending weight may vary between 100% and 0% within the corresponding first blending region 331a.

[0095] According to some embodiments herein, the variation of the blending weights is selected or determined such that the sum of pixel values ​​in the blending region remains the same for the first and second stitched frames. For example, the sum of the blending weights for pixel location p1 may be constant between the first stitched image frame 321 and the second stitched image frame 322. If the second blending weight for pixel location p1 is increased from 0% to 10% compared to the first blending weight, a further second blending weight for the same pixel location p1 may be decreased from 100% to 90%.

[0096] Therefore, the resulting pixel value of the first stitched frame 321 may be the same or nearly the same as the resulting pixel value of the second stitched frame 322 for the same pixel location.

[0097] action 404 In order for the method to eliminate pixel variations due to the movement of moving objects in the scene, at least one static portion 334 of the blend region 331a, 331b may be determined and / or selected. The static portion 334 may also be referred to as a patch.

[0098] At least one static portion 334 may form part of the first blending region 331a of the first and further first image frames 311a, 311b that is invariant with respect to the second blending region 331b of the second and further second image frames 312a, 312b if the second blending function 333b is not different from the first blending function 333a. Invariant in the embodiments herein includes no variation at all, but may also include having small variations less than or equal to the noise level of the original image frames.

[0099] To avoid complicating the computation of the difference pattern, the static portion 334 may be selected from among the blending regions 331a, 331b where there are positive blending weights for both blended image frames.

[0100] As mentioned above, embodiments herein allow for simplifying the evaluation of complex non-linear misalignment errors in stitched image frames 321, 322, such as rotations of the original images 311a, 311b, 312a, 312b being stitched, by evaluating the misalignment as a local translational misalignment in many small static portions 334 spread over the blending regions 331a, 331b, thereby reducing the complexity of the misalignment error calculation.

[0101] Thus, in some embodiments, at least one static portion 334 is composed of multiple static portions 334a, 334b, such as a first static portion 334a and a second static portion 334b. Additionally, three, or four, or even more static portions are possible. Each static portion 334a, 334b ​​may be composed of a small number of pixels, such as 8x8 or 16x16 pixels. The size of the static portion, whether multiple static portions or a single static portion, may be limited to certain size limits. For example, the size of the static portion may be limited to allow for linearization of complex alignment errors into linear alignment errors.

[0102] Also, limitations on the processing power of the image processor may impose a limit on the size of the static portion. However, there is a trade-off with the size of the static portion with respect to being able to meaningfully determine the error against any noise. A larger static portion means more statistical points are available to average out the noise. Furthermore, the static portion 334 must be larger than the linear alignment error to be determined.

[0103] As mentioned above, having multiple static portions can improve the assessment of the alignment. Multiple static portions 334a, 334b ​​can be advantageous compared to a single static portion 334 when the misalignment is due to rotation or skew. For example, by using multiple static portions 334a, 334b ​​to assess the alignment, the assessment can include information about multiple alignment degrees of freedom, such as linear alignment in two directions, and / or rotation, and / or skew. For example, the assessment of each of the multiple static portions 334a, 334b ​​may constitute different respective assessment results, such as different respective scales of the difference pattern, or different signs of the difference. By analyzing the difference patterns from the multiple static portions 334a, 334b, a conclusion can be drawn based on different scales and / or signs of the multiple difference patterns. For example, different signs of the difference patterns from two static portions located on different sides of the centerline may indicate that rotation is causing the misalignment. In this manner, a composite assessment of the different respective measures of each of the multiple static portions 334a, 334b ​​may improve the assessment of the misalignment compared to an assessment of a single static portion. As discussed above, by having multiple small static portions 334a, 334b, complex misalignment errors can be assessed based on an assessment of the linear translational misalignment in each less complex static portion 334a, 334b.

[0104] Furthermore, for some misalignments, such as rotational misalignments, the magnitude of the minimum misalignment error that can be determined in the embodiments herein may depend on the distance between the pixels used for evaluation. For example, the minimum misalignment error that can be determined may depend on the distance between the first static portion 334a and the second static portion 334b. In the scenario, a small rotation, such as 0.1 degrees, is assumed.

[0105] Furthermore, it is assumed that the size of the static parts, whether multiple static parts or a single static part, is limited to a certain size limit, for example, due to the limited processing power of the image processing device. For example, the size of the static parts 334a, 334b ​​may be limited to 16x16 pixels. Thus, each static part 334a, 334b ​​may be composed of 16x16 pixels. In this scenario, there is a great advantage in using multiple static parts 334a, 334b ​​with some spacing instead of a single static part 334. The distance between the multiple static parts 334a, 334b ​​may be selected based on the size of the original image frames 311a, 311b, 312a, 312b and the number of multiple static parts 334a, 334b ​​used. For example, the distance may be selected so that the area in which the multiple static parts 334a, 334b ​​are distributed is maximized.

[0106] To find a 0,1 degree rotational misalignment in a single static portion of 16x16 pixels, embodiments herein may need to be able to detect misalignments within a single static portion 334 on the order of a few hundredths of a pixel. Alternatively, if there is a distance of 500 pixels between the static portions 334a, 334b, embodiments herein may need to be able to detect misalignments between the static portions 334a, 334b ​​on the order of 1 pixel. Detecting misalignments between the static portions 334a, 334b ​​may be performed by comparing the absolute translation error difference between the two static portions 334a, 334b. Methods for determining such translation errors are briefly described below in operation 406. Additionally, the use of multiple static portions 334a, 334b ​​allows for a statistical measure of the difference pattern to be used in assessing alignment quality, which can reduce the effects of noise.

[0107] operation 405 The evaluation of the first alignment is performed based on a difference pattern based on the first stitched frame 321 and the second stitched frame 322. The image processing device 110 may obtain the difference pattern to evaluate the first alignment. The image processing device 110 may obtain the difference pattern by calculating the difference pattern.

[0108] The difference pattern is calculated based on corresponding pixel values ​​of the first stitched frame 321 and the second stitched frame 322. The corresponding pixels relate to the same pixel locations in the first and second stitched image frames 321, 322. The pixels of the first stitched frame 321 and the second stitched frame 322 are configured within at least one static portion 334 of the first and second blending regions 331a, 331b in the first and second stitched image frames 321, 322, respectively. Thus, the difference pattern is based on the at least one static portion 334.

[0109] In some embodiments, the difference pattern is calculated based on the difference between corresponding pixel values ​​of the first stitched frame 321 and the second stitched frame 322.

[0110] In some other embodiments, the difference pattern is calculated based on the difference between weighted values ​​of corresponding pixels of the first stitched frame 321 and the second stitched frame 322.

[0111] The difference pattern may be composed of a pixel difference measure of at least one static portion 334 of the blending regions 331a, 331b. In other words, the pixel difference measure may be a measure of difference between the first blending region 331a and the second blending region 331b.

[0112] In some embodiments, the pixel difference measure is based on a difference based on corresponding pixel values ​​of the first stitched frame 321 and the second stitched frame 322. For example, the pixel difference measure may be based on a difference between corresponding pixel values ​​of the first stitched frame 321 and the second stitched frame 322. The difference between corresponding pixel values ​​of the first stitched frame 321 and the second stitched frame 322 may be a difference between weighted values. In other words, the pixel difference measure may be based on weighted values ​​of corresponding pixels of the first stitched frame 321 and the second stitched frame 322.

[0113] In some other embodiments in which the assessment of the first alignment includes a reconstruction of the original image frames 311a, 311b, 312a, 312b, the reconstruction may be based on a pixel difference measure. For such embodiments, the assessment of the first alignment may further be based on a further difference pattern based on differences between corresponding pixel values ​​of the reconstructions of the first image frame 311a and the further first image frame 311b in the at least one static portion 334.

[0114] As mentioned above, the assessment of the first alignment may be further based on feature matching of the original image frames. In some embodiments, a further difference pattern may be based on the feature matching. For example, a further measure of pixel difference may be based on the difference between corresponding pixel values ​​of the reconstructions of the first image frame 311a and the further first image frame 311b. For all the examples of pixel difference calculations above, the measure of pixel difference may be:

[0115] A value describing the magnitude of the pixel difference, for example in the same range as used for the original pixel values ​​of the original image frames 311a, 311b, 312a, 312b, for example in the range 0 to 255. The magnitude of the pixel difference can for example be used when evaluating multiple alignments, in other words when evaluating multiple sets of alignment parameters. The magnitudes of the different alignments can then be compared and the smallest pixel difference can be selected.

[0116] A vector, e.g., describing the magnitude and direction of pixel differences, e.g., in which direction the pixel differences increase. Such an embodiment is well suited for use with a neural network to find suitable alignment parameters from a single difference pattern, also described below in operation 408.

[0117] The difference of single pixel values ​​of the first and second stitched image frames 321, 322, e.g. the pixel difference measure may consist of a respective pixel difference measure for each pixel. For example, the pixel difference measure may consist of a value of 3 for the first pixel, a value of 5 for the second pixel, while the pixel difference measure for other pixels is a difference of zero.

[0118] Weighted pixel value based difference,

[0119] A statistical measure such as the average of pixel differences of the first and second stitched image frames 321, 22 over the static portion 334;

[0120] a statistical measure such as an average of pixel differences of the first and second stitched image frames 321, 322 across the multiple static portions 334a, 334b;

[0121] Any combination of the above examples; may include.

[0122] The difference pattern is used below in operation 406 as a measure of how good the alignment is, in other words as a measure of the alignment error. As mentioned above, the difference pattern of a particular static portion 334a, 334b ​​may be used to determine a measure of the local alignment error.

[0123] operation 406 The image processor 110 evaluates the difference pattern to determine if realignment is necessary.

[0124] In some first embodiments, the image processor 110 evaluates the first alignment and, if it is not satisfactory, continues to evaluate other alignments. For example, other parameters of the camera for mapping the original image to the stitched image are selected and further evaluation of these alignments is performed in a trial and error manner. For example, the difference pattern of the first alignment may be compared to the difference patterns of one or more further alignments, such as the second alignment. The alignment that results in the smallest difference pattern may then be selected. In those embodiments, the difference pattern may be calculated by calculating the difference between pixel values ​​of the two stitched images, as also described below in operation 409.

[0125] In some second embodiments, the image processing device 110 evaluates the first alignment and, based on the results, determines an improved, preferably optimal, alignment, for example using a pre-trained neural network. For example, a difference pattern, such as a single difference pattern, is used as an input to the neural network along with one of the stitched images 321, 322 used to generate the difference pattern. After initial training, the neural network can evaluate the alignment based on the difference pattern and the stitched images 321, 322, as also described below in operation 408.

[0126] In some third embodiments, the difference pattern is used to calculate a respective reconstruction of the original images 311a, 311b or 312a, 312b in the static portion 334 together with one of the stitched images 321, 322. Then, evaluating the first alignment based on the difference pattern includes calculating a respective reconstruction of the first image frame 311a and the further first image frame 311b based on the difference pattern and further based on corresponding pixel values ​​of either the first stitched frame (321) or the second stitched frame (322). The calculation is performed for pixels in at least one static portion 334. The evaluation may further include comparing the respective reconstructions with each other. In other words, the respective reconstructions are based on corresponding pixel values ​​of the first stitched frame 321 and the second stitched frame 322 in the blending regions 331a, 331b, and further based on the first blending function 333a and the second blending function 333b. Since the static portion 334 is assumed to be unchanged between the first image frame 311a and the second image frame 312a, the reconstructed first image frame of the first image stream 310a should be composed of the same pixel values ​​as the reconstructed second image frame of the first image stream 310a. This means that the assessment of the first alignment may be further based on corresponding pixel values ​​of the respective reconstructions of the second image frame 312a and the further second image frame 312b.

[0127] As described above in operation 405, the difference pattern used to reconstruct the original image frame may be calculated based on the differences between weighted values ​​of corresponding pixels in the first stitched frame 321 and the second stitched frame 322.

[0128] An advantage of evaluating the first alignment further based on pixel values ​​of the reconstructed image frames is that such evaluation may be based on any known method for extracting differences and / or similarities between these original images, for example by comparing and / or matching the first and further first images 311a, 311b or the second and further second images 312a, 312b. As mentioned above, such matching methods may include SSD, NCC, or SIFT. Such algorithms may also be used to adjust the alignment if the first alignment is not good enough.

[0129] The following example outlines the details of how to evaluate the first alignment based on a reconstruction of the original images 311a, 311b or 312a, 312b. Let R1 be the right pixel value of the first image frame 311a in the static portion 334 of the first blending region 331a. L1 is the corresponding left pixel value of the further first image frame 311b in the static portion 334 of the first blending region 331a. The following assumes blending with alpha blending as described above: a is the blend weight for the right pixel value R1; (a-1) is the blend weight for the left pixel value L1; C1 is the pixel value of the first stitched image frame 321 corresponding to pixel values ​​R1 and L1. C2 is the pixel value of the second stitched image frame 322 corresponding to pixel values ​​R2 and L2, where R2 is the right pixel value of the second image frame 312a in the static portion 334 of the second blending region 331b, and L2 is the corresponding left pixel value of the further second image frame 312b in the static portion 334 of the second blending region 331b. Assume further that the blending weights are adjusted +10% for R2 to (a+0,1) for the second image frame 312a, and the blending weights are adjusted accordingly -10% for L2 to (1-a-0,1) for the further second image frame 312b. Since the pixel positions are configured in the static portion 334, it can be assumed that R1==R2, L1==L2. This results in the following linear equation system:

[0130] a * R1+(1-a) * L1=C1 (first blend) (a+0.1) * R1+(0.9-a) * L1=C2 (second blend)

[0131] By solving a system of linear equations, R1 and L1 can be determined for each pixel of the static portion 334. The blending weight a is a function of the pixel position. This means that the pixel value of the original blended image in the static portion 334 can be calculated or reconstructed. The calculation of R1 and L1 is based on a difference pattern based on the corresponding pixel values ​​of the first and second stitched image frames 321, 322. The difference pattern may be calculated based on the difference between the corresponding pixel values ​​of the first stitched frame (321) and the second stitched frame (322). For example, R1 and L1 can be calculated as follows: C1=a * R1+(1-a) * L1) and the pixel values ​​of the first stitched image frame 321 given by difference image C1-C2=0,1 * (R1-L1). As can be seen, the difference values ​​C1-C2 are described by the same formula for all other pixel positions. Since the alpha blending results in a difference image C1-C2 that is directly dependent on R1 and L1, this blending method allows for an accurate reconstruction of the original image in the static portion 334 based on the above.

[0132] The area covered by both the first and second blending regions 331a, 331b can be used to calculate the original image frame. The advantage of using said region for reconstruction is that within this region the original image frame is reconstructed with the same accuracy.

[0133] As mentioned above, by reconstructing the original images, known algorithms for feature identification, description and matching, such as SSD, can be used to match features in the corresponding original image frames, specifically in the portions of the original image frames corresponding to the static portions 334a, 334b. For example, a first feature in the first image frame 311a can be matched with a corresponding further first feature in the further first image frame 311b. Similarly, a second feature in the second image frame 311a can be matched with a corresponding further second feature in the further second image frame 312a. In a further example of evaluating the alignment based on the reconstructed original images, the alignment error may be +2 pixels in a first direction and -3 pixels in a second perpendicular direction for a particular static portion 334a. After possible averaging over the static portions 334a, 334b, a small adjustment of the alignment can be made in the direction of the derivative or derivative of the alignment error. The evaluation of the new alignment can then be compared to the old alignment.

[0134] Thus, in all the above embodiments, the image processor 110 evaluates the first alignment based on a difference pattern calculated based on pixel values ​​of the first stitched frame 321 and the second stitched frame 322. As described above in describing operation 405, the difference pattern is based on at least one static portion 334. That is, the pixels of the first stitched frame 321 and the second stitched frame 322 are configured within at least one static portion 334 of the first and second blend regions 331a, 331b in the first and second stitched image frames 321, 322, respectively.

[0135] In some embodiments herein, the image processing device 110 evaluates the first alignment by calculating a difference pattern. For example, the image processing device 110 may evaluate the first alignment by determining an alignment degree of the first alignment based on the difference pattern. The alignment degree is an indicator of whether the alignment is good or bad. In other words, the alignment degree may be a measure of alignment quality, or may be a measure of misalignment or misalignment error.

[0136] Evaluating the first alignment based on the difference pattern may further include comparing the difference pattern to a threshold or to a predetermined difference pattern. For example, the threshold may be a threshold average value of pixel differences of the first and second stitched image frames 321, 322 over the static portion 334. The threshold may be related to a noise level of the stitched images 321, 322. That is, if the difference pattern exceeds the threshold, it may be determined that the difference pattern is not simply noise.

[0137] In some embodiments, evaluating the first alignment includes calculating a measure of alignment quality, such as a degree of alignment that increases with a decrease in the difference pattern, which may correspond to a decrease in the absolute value of the pixel differences.

[0138] The degree of alignment may be determined by comparing the difference pattern to a threshold. For example, if the difference pattern is greater than zero, the alignment is determined to be incomplete. The method may then continue by selecting a new alignment and evaluating the new alignment based on the new difference pattern, for example by determining the degree of alignment of the new alignment, as described below in operation 407.

[0139] However, if the difference pattern is zero or close to zero, it may be determined that there is no problem with the alignment and the evaluation may be terminated.

[0140] As mentioned above, if the images are well aligned, blending variations should have minimal effect on the static portions of the resulting stitched image, so the difference pattern is a good measure of the degree of alignment or misalignment, alignment quality, or misalignment measure, of the images being stitched.

[0141] The advantage of basing the assessment or determination of the degree of alignment or alignment quality on the difference pattern of the static portion 334 is that the assessment or determination of a correctly aligned stitched image is transparent to the rest of the imaging system and / or the end user because the difference pattern is zero for a perfectly aligned image. If the images are not perfectly aligned, the stitched image is only slightly affected by the assessment. However, stitching of images that are not perfectly aligned will in any case produce a stitched image that has stitching errors due to misalignment, and there is a need for realignment. Using the embodiments herein, it is possible to realign misaligned images without affecting correctly aligned images and without using a separate image stream to assess alignment.

[0142] In some embodiments herein, the calculation or determination of the misalignment error is based on the difference pattern. As described above in operation 405, the measure of pixel difference may be composed of, for example, vector values ​​including directional information. In some embodiments, the calculation and / or determination may be performed by a neural network. For example, a pre-trained neural network may be used to determine the measure of alignment error based on the difference pattern in the blend region 331a and the original stitched images 321, 322. For example, the pre-trained neural network may use two input tensors, one for the stitched images 321, 322 and another for the difference pattern, in other words, the difference image. Based on the determined alignment error, an improved alignment may be determined according to known methods. For example, for the stitched images 321, 322, there may be different alignment errors, for example corresponding to linear alignment errors in two directions, and / or rotational errors, and / or skew. The determination of the misalignment error may include determining both the magnitude of the error and the direction of the error, for example, the direction in the plane of the image. If the alignment error is determined to be linear along an axis in the plane of the images, then the stitched images can be linearly adjusted in an amount and direction based on the misalignment error. For example, if the alignment error is 5 pixels in one direction, the stitched images can be realigned by remapping the pixels 5 pixels in the opposite direction of the alignment error so that the alignment error disappears.

[0143] An advantage of the embodiments herein is that the assessment of alignment is based on information embedded in the stitched image stream 310c, and therefore does not require a separate image stream for assessment.

[0144] A further advantage of the embodiments herein is that it is possible to assess alignment from fewer images, e.g., images stitched by varying the blend instead of comparing the stitched image with the original images, e.g., by determining the misalignment error.

[0145] A further advantage is that complex alignment errors can be modeled by simpler linear alignment errors of each static portion 334.

[0146] Although the embodiments are described above as determining a local translation error within each static portion 334a, 334b, the embodiments are also applicable to other alignment errors, such as rotational alignment errors. For example, a rotational error may be determined based on many local translation errors within each static portion 334a, 334b. Thus, expressing the direction of the error also includes the rotational direction.

[0147] behavior 407 If the difference pattern is above the threshold, the method may further include determining or selecting a second alignment. The determination or selection of the second alignment may be performed to use the second alignment without further evaluation, or to evaluate the second alignment in a corresponding manner as the first alignment was evaluated. The above operations 401a and 401b may then be repeated corresponding to the second alignment. Thus, the image processing device 110 may align the first image frame 311a and the further first image frame 311b according to the second alignment, and align the second image frame 312a and the further second image frame 312b according to the second alignment. The alignment may be performed to evaluate the second alignment, for example, by determining the degree of alignment of the second alignment.

[0148] However, in order to allow uninterrupted use of the above images and image streams in the video network system 250, the first image frame 311a and the further first image frame 311b are generally not the same as those used for the first alignment. Instead, new image frames can be used for the evaluation of the second alignment. Correspondingly, the second image frame 312a and the further second image frame 312b used for the evaluation of the second alignment are generally not the same as the corresponding frames used for the first alignment.

[0149] behavior 408 In some embodiments, the image processor 110 selects an alignment from among the multiple alignments based on evaluating the first alignment, e.g., based on a difference pattern or a measure thereof. In some other embodiments, the image processor 110 determines the alignment based on evaluating the first alignment, e.g., based on a difference pattern or a measure thereof.

[0150] Selecting the alignment may include selecting the above-mentioned parameters of the video camera 325, such as parameters related to the mounting positions of the sensors 301a, 301b. The alignment selection may also be performed by a neural network, etc. How such a selection is performed based on the difference pattern is beyond the scope of the embodiments herein.

[0151] As an intermediate operation, the evaluation can be based on the determined degree of alignment of the first alignment, which can be based on the difference pattern or a measure thereof. For example, if the image processing device 110 determines that the quality or degree of alignment of the first alignment is sufficiently good, for example, by determining that the difference pattern is below a threshold, the image processing device 110 can decide to use the first alignment of the multiple alignments. On the other hand, if the image processing device 110 determines that the degree of alignment of the first alignment is not sufficiently good, for example, by determining that the difference pattern is above a threshold, the image processing device 110 can decide to use an alignment different from the first alignment of the multiple alignments. The image processing device 110 can decide to use the second alignment instead, for example.

[0152] operation 409 In some embodiments, the image processor 110 evaluates multiple alignments to determine which is the optimal alignment. Operations 401-406 above may then be repeated as described above in relation to operation 407 where a second alignment is determined or selected. Correspondingly, a third alignment, etc. may also be determined, evaluated and compared to the other alignments.

[0153] For example, if the second alignment is determined or selected, the image processing device 110 may evaluate the second alignment by determining the degree of alignment of the second alignment based on, for example, a second difference pattern calculated based on pixel values ​​of the first stitched image frame 321 obtained using the second alignment and the first blending function 333b and pixel values ​​of the second stitched image frame 322 obtained using the second alignment and the second blending function. The determination of the degree of alignment of the second alignment may be performed according to operation 406 above. Further alignments may be evaluated in a corresponding manner. In some embodiments, the static portion used for the calculation of the pixel difference measure is the same for the second alignment as for the first alignment. However, a new static portion, such as a second static portion different from the at least one static portion 334, may also be used for the calculation of the pixel difference measure.

[0154] A neural network can be trained to evaluate the effects of blending, iteratively testing and evaluating new alignments of the images to find the best alignment, e.g., the alignment associated with the smallest change, e.g., the alignment that results in the smallest measure of the difference pattern.

[0155] operation 410 When multiple alignments have been evaluated according to the above description of operation 409, an alignment selection may be performed to select a best or optimal alignment from among the multiple alignments. The selection may be based on the smallest difference pattern among the multiple difference patterns corresponding to the respective alignments, i.e., based on which difference pattern is smallest.

[0156] For example, when the image processing device 110 evaluates first and second alignments, the image processing device 110 can select an alignment from the first alignment and the second alignment based on the smallest difference pattern among the first difference pattern and the second difference pattern.

[0157] In some embodiments, the smallest difference pattern of the first and second difference patterns consists of the smallest difference between the first blending region 331a and the second blending region 331b. That is, the smallest difference pattern of the first and second difference patterns may consist of the smallest difference between the static portion 334 of the first blending region 331a and the static portion 334 of the second blending region 331b. For example, the sum of pixel differences for all static portions 334a, 334b ​​and / or the sum of average pixel differences may be compared for each alignment.

[0158] FIG. 5 shows an exemplary graph of a measure of the difference pattern, called difference, for different alignment options, called alignment nr. As can be seen, alignment option 6 corresponds to zero or at least a very small misalignment, since the difference is zero for alignment nr6. The other alignment options correspond to different misalignments, for example linear misalignments along the axis of the image frame. Since the magnitude of the measure of the difference pattern varies depending on the misalignment, it is possible to predict the misalignment based on the difference pattern, and possibly also based on information about the direction of the misalignment. The direction information can be obtained, for example, from a calculation of the derivative or difference of the difference pattern, such as a difference pattern based on two or more alignments.

[0159] 6, there is shown a schematic block diagram of an embodiment of an image processing device 600. The image processing device 600 is configured to evaluate a first alignment, i.e., an alignment of a first image stream 310a from a first image sensor 301a with a second image stream 310b from a second image sensor 301b. As mentioned above, the image processing device 600 may include an image capture device 110, such as a video camera 325, a surveillance camera, a network video recorder, or a wireless communication device.

[0160] The image processing device 600 may comprise a processing module 601 such as means for carrying out the methods described herein, which may be embodied in the form of one or more hardware modules and / or one or more software modules.

[0161] The imaging device 600 may further include a memory 602. The memory may include or store instructions, for example in the form of a computer program 603, which, when executed on the imaging device 600, may constitute computer readable code units that cause the imaging device 600 to perform a method.

[0162] According to some embodiments herein, the image processing device 600 and / or the processing module 601 comprises a processing circuit 604 as an example hardware module that may include one or more processors. Thus, the processing module 601 may be embodied or "implemented" in the form of the processing circuit 604. Instructions may be executable by the processing circuit 604, such that the image processing device 600 is operable to perform the method of FIG. 4. As another example, the instructions, when executed by the image processing device 600 and / or the processing circuit 604, may cause the image processing device 600 to perform the method according to FIG. 4.

[0163] Hence, in one example, an image processing device 600 is provided for evaluating a first alignment, i.e. an alignment of a first image stream 310a from a first image sensor 301a and a second image stream 310b from a second image sensor 301b. Again, the memory 602 includes instructions executable by said processing circuitry 604, whereby the image processing device 600 is operable to perform the method according to FIG.

[0164] A first image frame 311a of a first image stream 310a from a first image sensor 301a and a further first image frame 311b of a second image stream 310b from a second image sensor 301b are aligned according to a first alignment, and a second image frame 312a of the first image stream 310a and a further second image frame 312b of the second image stream 310b are aligned according to the first alignment.

[0165] A first stitched image frame 321 is obtained by blending the aligned first image frame 311a and the further first image frame 311b in a first blending region 331a of the first and further first image frames 311a, 311b according to a first blending function, and a second stitched image frame 322 is obtained by blending the aligned second image frame 312a and the further second image frame 312b in a second blending region 331b of the second and further second image frames 312a, 312b according to a second blending function different from the first blending function.

[0166] The first alignment is evaluated based on a difference pattern calculated based on pixel values ​​of the first stitched frame 321 and the second stitched frame 322. The difference pattern is based on at least one static portion 334 of the first and second blend regions 331a, 331b in each of the first and second stitched image frames 321, 322.

[0167] The image processing device 600 is further operable to perform the method according to the detailed embodiment described above in relation to FIG.

[0168] 6 further shows a carrier 605, i.e. a program carrier, constituting the computer program 603 as just described. The carrier 605 may be one of an electronic, optical, or radio signal and a computer readable medium.

[0169] In some embodiments, image processing unit 600 and / or processing module 601 may be comprised of one or more of the following exemplary hardware modules: alignment module 610, acquisition module 620, evaluation module 630, and selection module 640. In other examples, one or more of the aforementioned exemplary hardware modules may be implemented as one or more software modules.

[0170] Furthermore, the processing module 601 may comprise an input / output unit 606. According to an embodiment, the input / output unit 606 may comprise a first and a second image sensor 301a, 301b configured to capture the above mentioned image frames, such as the first image frame 311a and the further first image frame 311b.

[0171] Thus, the image processing device 600 is configured to assess the alignment of the first image stream 310a from the first image sensor 301a and the second image stream 310b from the second image sensor 301b.

[0172] Therefore, according to the various embodiments described above, the image processing device 600, and / or the processing module 601, and / or the alignment module 6610 are configured to align a first image frame 311a of the first image stream 310a from the first image sensor 301a with a further first image frame 311b of the second image stream 310b from the second image sensor 301b in accordance with the first alignment, and further to align a second image frame 312a of the first image stream 310a with a further second image frame 312b of the second image stream 310b in accordance with the first alignment.

[0173] The image processing device 600, and / or the processing module 601, and / or the alignment module 610 may be further configured to align the first image frame 311a and the further first image frame 311b in accordance with the second alignment, align the second image frame 312a and the further second image frame 312b in accordance with the second alignment, and evaluate the second alignment, for example by determining a degree of alignment of the second alignment.

[0174] The image processing device 600, and / or the processing module 601, and / or the acquisition module 620 are configured to obtain a first stitched image frame 321 by blending the aligned first image frame 311a and the further first image frame 311b in a first blending region 331a of the first and further first image frames 311a, 311b according to a first blending function 333a, and further obtain a second stitched image frame 322 by blending the aligned second image frame 312a and the further second image frame 312b in a second blending region 331b of the second and further second image frames 312a, 312b according to a second blending function 333b different from the first blending function 333a.

[0175] The image processing device 600, and / or the processing module 601, and / or the evaluation module 630 may be configured to evaluate the first alignment based on a difference pattern calculated based on pixel values ​​of the first stitched frame 321 and the second stitched frame 322. The difference pattern is based on at least one static portion 334 of the first and second blending regions 331a, 331b in each of the first and second stitched image frames 321, 322.

[0176] The evaluation may be performed by determining a degree of alignment of the first alignment based on the difference pattern.

[0177] The difference pattern may be calculated based on the difference between corresponding pixel values ​​of the first stitched frame 321 and the second stitched frame, or in some other embodiments, the difference between weighted values ​​of corresponding pixels of the first stitched frame 321 and the second stitched frame 322.

[0178] In some embodiments, evaluating the first alignment based on the difference pattern includes calculating respective reconstructions of the first image frame 311a and the further first image frame 311b in the at least one static portion 334 based on the difference pattern and further based on corresponding pixel values ​​of either the first stitched frame 321 or the second stitched frame 322, and then comparing the respective reconstructions with each other.

[0179] The image processing device 600, and / or the processing module 601, and / or the selection module 640 may be configured to select an alignment from among the multiple alignments based on evaluating the first alignment. In some embodiments, the image processing device 600, and / or the processing module 601, and / or the selection module 640 are configured to select an alignment from among the first alignment and the second alignment based on a minimum difference pattern among the first and second difference patterns.

[0180] The term "module" as used herein may refer to one or more functional modules, each of which may be implemented as one or more hardware modules, and / or one or more software modules, and / or a combined software / hardware module. In some embodiments, a module may represent a functional unit that is implemented as software and / or hardware.

[0181] As used herein, the terms "computer program carrier," "program carrier," or "carrier" may refer to one of an electronic signal, an optical signal, a radio signal, and a computer-readable medium. In some examples, the computer program carrier may exclude transitory propagating signals, such as electronic signals, optical signals, and / or radio signals. Thus, in these examples, the computer program carrier may be a non-transitory carrier, such as a non-transitory computer-readable medium.

[0182] The term "processing module" as used herein may include one or more hardware modules, one or more software modules, or a combination thereof. Any such module may be hardware, software, or a combined hardware-software module, and may be an alignment means, an acquisition means, an evaluation means, a selection means, etc., as described herein. As an example, the reference to "means" may be a module corresponding to the modules listed above in connection with the drawings.

[0183] As used herein, the term "software module" may refer to a software application, a dynamic link library (DLL), a software component, a software object, an object conforming to the Component Object Model (COM), a software component, a software function, a software engine, an executable binary software file, or the like.

[0184] The terms "processing module" or "processing circuitry" as used herein may encompass a processing unit including, for example, one or more processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. A processing circuitry or the like may include one or more processor kernels.

[0185] As used herein, the phrase "configured / configured for" may mean that a processing circuit is configured, e.g., adapted or operable, to perform one or more operations described herein, by a software configuration and / or a hardware configuration.

[0186] As used herein, the term "operation" may refer to an action, a step, an operation, a response, a reaction, an activity, etc. It should be noted that an operation herein may be divided into two or more sub-operations, if desired. Furthermore, it should be noted that two or more operations described herein may be merged into a single operation, if desired.

[0187] As used herein, the term "memory" may refer to a hard disk, a magnetic storage medium, a portable computer diskette or disk, a flash memory, a random access memory (RAM), etc. Additionally, the term "memory" may refer to a processor's internal register memory, etc.

[0188] The term "computer-readable medium" as used herein may be a Universal Serial Bus (USB) memory, a DVD disk, a Blu-ray disk, a software module received as a stream of data, a flash memory, a hard drive, and a memory card such as a Memory Stick, a Multimedia Card (MMC), or a Secure Digital (SD) card. One or more of the foregoing examples of computer-readable medium may be provided as one or more computer program products.

[0189] As used herein, the term "computer readable code unit" may refer to the text of a computer program, all or part of a binary file representing a computer program in compiled form, or anything in between.

[0190] As used herein, the terms "number" and / or "value" may refer to any type of numerical value, such as a binary number, a real number, an imaginary number, or a rational number. Furthermore, a "number" and / or "value" may be one or more characters, such as a letter or a string of characters. A "number" and / or "value" may also be represented by a string of bits, i.e., 0 and / or 1.

[0191] As used herein, the phrase "in some embodiments" is used to indicate that features of the described embodiments can be combined with other embodiments described herein.

[0192] Although embodiments of various aspects have been described, many different changes, modifications, etc. thereof will become apparent to those skilled in the art. Therefore, the described embodiments are not intended to limit the scope of the present disclosure.

Claims

1. A method performed by an image processing device (110, 600), comprising: evaluating an alignment of a first image stream (310a) from a first image sensor (301a) with a second image stream (310b) from a second image sensor (301b), the first and second image sensors (301a, 301b) having overlapping fields of view, the method comprising: - aligning (401a) a first image frame (311a) of the first image stream (310a) with a further first image frame (311b) of the second image stream (310b) according to a first alignment; aligning (401b) a second image frame (312a) of the first image stream (310a) with a further second image frame (312b) of the second image stream (310b) according to the first alignment; obtaining (402) a first stitched image frame (321) by blending (311a) the aligned first image frame (311a) and the further first image frame (311b) in a first blending region (331a) of the first and further first image frames (311a, 311b) according to a first blending function (333a); and obtaining (403) a second stitched image frame (322) by blending (333b) the aligned second image frame (312a) and the further second image frame (312b) in a second blending region (331b) of the second and further second image frames (312a, 312b) according to a second blending function (333b), the method comprising: evaluating (406) the first alignment based on a difference pattern calculated based on corresponding pixel values ​​of the first stitched frame (321) and the second stitched frame (322), the corresponding pixels being included within at least one static portion (334) of the first and second blend regions (331 a, 331 b) in the respective first and second stitched image frames (321, 322), the method further comprising: the second blending function (333b) is different from the first blending function (333a); the first blending function (333a) defines blending weights of the aligned first image frame (311a) and the further first image frame (311b) as a function of pixel position within the first blending region (331a); The method, characterized in that the second blending function (333b) defines respective blending weights of the aligned second image frame (312a) and the further second image frame (312b) as a function of pixel position within the second blending region (331b).

2. 2. The method of claim 1, wherein the first and second image sensors (301a, 301b) are included in the same image capture device (110), the first image frame (311a) and the further first image frame (311b) are captured at a same or similar first time point, and the second image frame (312a) and the further second image frame (312b) are captured at a same or similar second time point that is different from the first time point.

3. The difference pattern is a. the difference between corresponding pixel values ​​of the first stitched frame (321) and the second stitched frame (322); and / or b. The difference between the weighted values ​​of corresponding pixels of the first stitched frame (321) and the second stitched frame (322). The method according to claim 1 or 2, wherein the calculation is based on

4. Evaluating the first alignment based on the difference pattern includes: calculating a reconstruction of each of the first image frame (311 a) and the further first image frame (311 b) in the at least one static portion (334) based on the difference pattern and further based on corresponding pixel values ​​of either the first stitched frame (321) or the second stitched frame (322); Comparing said respective reconstructions with each other. The method according to any one of claims 1 to 3, comprising:

5. The method of any one of claims 1 to 4, wherein evaluating the first alignment based on the difference pattern comprises comparing the difference pattern to a threshold.

6. The method of any one of claims 1 to 5, further comprising selecting (408) an alignment from among a plurality of alignments based on evaluating the first alignment.

7. The difference pattern exceeds a threshold, and the method further comprises: aligning (401a) the first image frame (311a) and the further first image frame (311b) according to a second alignment; aligning (401b) the second image frame (312a) and the further second image frame (312b) according to the second alignment to evaluate the second alignment. The method according to any one of claims 1 to 6, comprising:

8. Evaluating (406) the second alignment based on a second difference pattern calculated based on pixel values ​​of the first stitched image frame (321) obtained with the second alignment and the first blending function (333a) and pixel values ​​of the second stitched image frame (322) obtained with the second alignment and the second blending function (333b); selecting (410) an alignment from among the first alignment and the second alignment based on a minimum difference pattern of the first and second difference patterns; The method of claim 7, comprising:

9. 9. The method of claim 8, wherein the minimum difference pattern of the first and second difference patterns comprises a minimum difference between the static portion (334) of the first blending region (331 a) and the static portion (334) of the second blending region (331 b).

10. The method according to any one of the preceding claims, comprising executing said method at run-time on said image processing device (110, 600).

11. An image processing device (110, 600) configured to perform the method according to any one of the preceding claims.

12. 12. The image processing apparatus (110, 600) of claim 11, comprising an image capture device (110) such as any of a camera, a surveillance camera, a monitoring camera or camcorder, a network video recorder, and a wireless communication device.

13. A computer program (603) comprising computer readable code units which, when executed on an image processing device (110, 600), causes said image processing device (110, 600) to perform the method according to any one of the preceding claims.

14. A carrier (605) containing the computer program of claim 13, the carrier (605) being one of an electronic signal, an optical signal, a radio signal, and a computer readable medium.

Images