Method for determining lens distortion correction of interchangeable camera lens and image processing apparatus

Abstract

A method for determining lens distortion correction includes: - acquiring (503) an indication of a first feature and a first correction position of the first feature in a first corrected image frame, the first correction position being based on a first lens distortion correction at a first uncorrected position of the first feature in a first uncorrected image frame captured by a camera equipped with a first interchangeable lens; - detecting (504) a change in the lens on the camera; - acquiring (506) an indication of a second feature and a second uncorrected position of the second feature in a second uncorrected image frame captured by a camera equipped with a second lens; - matching the first feature with the second feature (507); and - determining (509) a second lens distortion correction of the second lens based on a mapping between the first correction position and the second uncorrected position.

Description

Technical Field

[0001] The embodiments herein relate to a method and image processing apparatus for determining lens distortion correction suitable for use with an interchangeable lens camera. Corresponding computer programs and computer program carriers are also disclosed. Background Technology

[0002] The use of imaging (especially video imaging) for public surveillance is common in many areas around the world. Areas that may need surveillance include banks, shops, and other areas requiring security, such as schools and government facilities. However, in many places, installing cameras without permission / authorization is illegal. Other areas that may need surveillance are processing, manufacturing, and logistics applications, where video surveillance is primarily used to monitor processes.

[0003] Due to limitations, all real lenses will produce some form of aberration, or image defect. This aberration occurs because the simple paraxial theory is not a completely accurate model of the effect of light on the optical system, rather than due to defects in the optical elements.

[0004] In optics, aberration is a characteristic of an optical system (such as a lens) that causes light rays to scatter across certain areas of space instead of focusing on a single point. Aberration results in blurred or distorted images formed by a lens, the nature of which depends on the type of aberration. Aberration can be defined as the deviation of the performance of an optical system from the predictions of paraxial optics. In an imaging system, this occurs when light from a point on an object fails to converge to a single point (or diverge from it) after traveling through the system.

[0005] The most common distortions are radially symmetrical, or nearly radially symmetrical, caused by the symmetry of the camera lens. These radial distortions can be classified, for example, as barrel distortion or pincushion distortion or a mixture of both, and are sometimes referred to as mustache distortion or complex distortion. Figure 1a The diagram schematically illustrates barrel and pincushion distortions that form the horizontal and vertical lines of a square. Mathematically, barrel and pincushion distortions can be described as quadratic distortions, meaning they increase with the square of the distance from the center.

[0006] Imaging optics systems with aberrations produce sharp images. Optical instrument manufacturers need to correct these aberrations. Lens distortion correction algorithms can be implemented by warping a distorted input image into a corrected, undistorted output image. This can be achieved by performing an inverse transform; for example, for each pixel (u, v) in the corrected output image, the corresponding coordinates (u, v) in the input image are calculated. ∨ v ∨ ).

[0007] Cameras equipped with lenses whose lens distortion correction algorithms are known can apply those algorithms to correct aberrations. This is, for example, the case for cameras equipped with default lenses.

[0008] There are methods for calculating the correction function for a camera lens. One example is capturing an image of an object with known straight lines, such as a chessboard. It is well known that the lines on a chessboard are straight, and the distortion correction function for distorted lines can be calculated using known methods. Another approach might be to guide the camera to an area in the scene that has some known straight lines. By matching lines in one or more distorted images, for example, with the chessboard, and with the straight lines in one or more corrected images, distortion correction can be calculated.

[0009] However, some cameras are designed for interchangeable lenses, or even interchangeable sensor heads that include lenses. That is, a camera can be adapted to work with different lenses. In this case, the camera may be able to use lens distortion correction algorithms for some lenses, but not all. This is problematic if the camera is equipped with a lens for which the correct lens distortion correction algorithm cannot be used. A further problem might be that the camera may not know which lens is mounted on it, and the camera may not even know that the lens has been changed. Therefore, the camera might apply a lens distortion correction algorithm to the other lens. For example, even though the camera is equipped with another lens, it might apply the lens distortion correction algorithm used for the default lens. Summary of the Invention

[0010] Therefore, the purpose of the embodiments described herein may be to eliminate some of the problems mentioned above, or at least reduce their impact. Specifically, the purpose of the embodiments described herein may be to correct lens distortion without a known lens distortion correction algorithm. In other words, the purpose of the embodiments described herein may be to calculate a lens distortion correction algorithm. Specifically, the purpose of the embodiments described herein may be to calculate a lens distortion correction algorithm for interchangeable lenses without using known models.

[0011] According to one aspect, the objective is achieved by a lens distortion correction method performed in an image processing device for determining an interchangeable camera lens suitable for use in an interchangeable lens camera.

[0012] The method includes obtaining an indication of a first feature and a first corrected position of the first feature in a first corrected image frame. The first corrected position is a first lens distortion correction based on a first uncorrected position of the first feature in a first uncorrected image frame, the first uncorrected image frame including the first feature and captured by the camera when equipped with a first interchangeable lens. The first uncorrected image frame captures a first scene.

[0013] The method also includes detecting changes in the lens on the camera from the first lens to the second interchangeable lens.

[0014] The method further includes obtaining an indication of a second feature corresponding to the first feature, and a second uncorrected position of the second feature in a second uncorrected image frame captured by the camera when equipped with the second lens. The second uncorrected image frame captures a second scene that at least partially overlaps with the first scene.

[0015] The method further includes matching the first feature with the second feature.

[0016] The method further includes determining a second lens distortion correction for the second lens based on a mapping between the first corrected position of the matched first feature and the second uncorrected position of the matched second feature of the lens.

[0017] In the embodiments described herein, the mapping between a first position in a first image frame and a second position in a second image frame implies a translation between the first and second positions, such as a translation from the first position to the second position. This translation can be accomplished, for example, using a function or table associated with the first and second positions.

[0018] According to another aspect, the objective is achieved by an image processing device configured to perform the above-described method.

[0019] According to another aspect, the stated objective is achieved by a computer program and a computer program carrier corresponding to the foregoing aspect.

[0020] The embodiments of this paper utilize the known first-lens distortion correction associated with the first lens to calculate the second-lens distortion correction for the second lens. The embodiments of this paper match the first feature with the second feature. Therefore, the first-lens distortion correction is used to calculate the "ground truth" to which the second feature in the second uncorrected image frame can be mapped.

[0021] Since the first lens distortion correction is used to calculate the "foundational facts," it is not necessary to capture images of known patterns, especially images containing lines known to be straight, such as in a checkerboard pattern, or fiducial markers. This reduces the complexity of obtaining the second lens distortion correction for the second lens.

[0022] The determination of the distortion correction for the second lens is real-time; that is, it takes into account the current state of the second lens, rather than using distortion correction calculated using sample lenses. Any scene including distinguishable features can be used. Any feature can be used, as long as it is possible to match the first feature captured with the first lens with the same feature captured with the second lens with a certain degree of determinism. No prior knowledge of the scene is required, nor is any special arrangement of the scene necessary. Attached Figure Description

[0023] Various aspects of the embodiments disclosed herein, including their specific features and advantages, will be readily understood from the following detailed description and accompanying drawings, wherein:

[0024] Figure 1a The diagram schematically illustrates barrel and pincushion distortions.

[0025] Figure 1b An exemplary embodiment of the image capture device is shown.

[0026] Figure 2a An exemplary embodiment of a video network system is shown.

[0027] Figure 2b Exemplary embodiments of a video network system and user equipment are shown.

[0028] Figure 3a This is a schematic block diagram illustrating an exemplary embodiment of the imaging system.

[0029] Figure 3b This is a schematic block diagram illustrating an exemplary embodiment of a sensor head including a lens.

[0030] Figure 3c This is a schematic block diagram illustrating an exemplary embodiment of a sensor head including a lens.

[0031] Figure 4 This is a schematic block diagram illustrating an embodiment of a method in an image processing apparatus.

[0032] Figure 5a This is a flowchart illustrating an embodiment of a method in an image processing device.

[0033] Figure 5b This is a schematic block diagram illustrating an embodiment of a method in an image processing apparatus.

[0034] Figure 5c This is a schematic block diagram illustrating an embodiment of a method in an image processing apparatus.

[0035] Figure 5d This is a schematic block diagram illustrating an embodiment of a method in an image processing apparatus.

[0036] Figure 6 This is a block diagram illustrating an embodiment of an image processing device. Detailed Implementation

[0037] The embodiments described herein can be implemented in one or more image processing devices. In some embodiments herein, the one or more image processing devices may include or be one or more image capture devices, such as digital cameras. Figure 1b Various exemplary image capture devices 110 are depicted. The image capture device 110 may be, for example, or include any of the following: a camcorder, a network video recorder, a camera, a video camera 120 (e.g., a surveillance camera or monitoring camera), a digital camera, a wireless communication device 130 including an image sensor (e.g., a smartphone), or a car 140 including an image sensor.

[0038] Figure 2a An exemplary video network system 250 is depicted therein, in which embodiments thereof may be implemented. The video network system 250 may include an image capture device, such as a video camera 120, which may capture digital images 201 (e.g., digital video images) and perform image processing thereon. Figure 2a The video server 260 can acquire images from the video camera 120, for example, via a network. Figure 2a The middle part is indicated by a double-pointing arrow.

[0039] Video server 260 is a computer-based device specifically designed for transmitting video. Video servers are used in many applications and often have additional features and capabilities to meet specific application needs. For example, video servers used in security, surveillance, and inspection applications are typically designed to capture video from one or more cameras and transmit that video over a computer network. In video production and broadcasting applications, video servers may be able to record and play back recorded video and transmit multiple video streams simultaneously. Many video server functionalities can now be built into video camera 120.

[0040] However, in Figure 2a In this embodiment, video server 260 is connected to an image capture device illustrated herein by way of video camera 120 via video network system 250. Video server 260 may also be connected to video storage device 270 for storing video images and / or to monitor 280 for displaying video images. In some embodiments, video camera 120 is directly connected to video storage device 270 and / or monitor 280, such as... Figure 2a The arrows between these devices are shown in the diagram. In some other embodiments, the video camera 120 is connected to the video storage device 270 and / or the monitor 280 via the video server 260, as shown by the arrows between the video server 260 and other devices.

[0041] Figure 2b A user equipment 295 is depicted connected to a video camera 120 via a video network system 250. The user equipment 295 may be, for example, a computer or a mobile phone. The user equipment 295 may, for example, control the video camera 120 and / or display video emitted from the video camera 120. The user equipment 295 may also include the functionality of a monitor 280 and a video storage device 270.

[0042] To better understand the embodiments described herein, the imaging system will be described first.

[0043] Figure 3a This is a schematic diagram of an imaging system 300, in this case a digital video camera, such as video camera 120. Generally, the imaging system 300 may be part of an image processing apparatus in which embodiments of this document are implemented. The imaging system 300 images a scene onto an image sensor 301. The image sensor 301 may be equipped with a Bayer filter such that different pixels will receive radiation in a specific wavelength region in a known pattern. Typically, each pixel of the captured image is represented by one or more values ​​that represent the intensity of light captured within a certain wavelength band. These values ​​are often referred to as color components or color channels. The term "image" may refer to an image frame or video frame that includes information derived from the image sensor that has captured the image.

[0044] After reading the signals from each sensor pixel of the image sensor 301, the image signal processor 302 can perform different image processing actions. The image signal processor 302 may include an image processing section 302a, sometimes referred to as an image processing pipeline, and a video post-processing section 302b.

[0045] Typically, for video processing, images are included in an image stream. Figure 3a A first video stream 310 from an image sensor 301 is shown. The first video stream 310 may include multiple captured image frames, such as a first captured image frame 311 and a second captured image frame 312.

[0046] Image processing may include desacrifice, color correction, noise filtering (for eliminating spatial and / or temporal noise), distortion correction (for eliminating effects such as barrel distortion), global and / or local tone mapping (e.g., enabling imaging of scenes with a wide range of intensities), transformations (e.g., correction and rotation), flat-field correction (e.g., for eliminating vignetting), applying overlays (e.g., privacy masks, explanatory text), etc. The image signal processor 302 may also be associated with an analysis engine that performs object detection, recognition, alerting, etc.

[0047] Image processing unit 302a may, for example, perform image stabilization, apply noise filtering, distortion correction, global and / or local tone mapping, transformation, and flat field correction. Video post-processing unit 302b may, for example, crop portions of the image, apply overlays, and include an analysis engine.

[0048] Following the image signal processor 302, the image can be forwarded to the encoder 303, where information in the image frame is encoded according to an encoding protocol such as H.264. The encoded image frame is then forwarded to, for example, a receiving client (monitor 280 in this case), a video server 260, a storage device 270, etc.

[0049] The video encoding process produces multiple values ​​that can be encoded to form a compressed bitstream. These values ​​may include:

[0050] • Quantization transformation coefficients,

[0051] • Enables the decoder to recreate the predicted information.

[0052] Information regarding the structure of the compressed data used in the encoding process and the compression tools, as well as

[0053] • Information about the complete video sequence.

[0054] These values ​​and parameters (syntactic elements) are converted into binary code using, for example, variable-length encoding and / or arithmetic encoding. Each of these encoding methods produces an efficient, compact binary representation of the information, also known as an encoded bitstream. The encoded bitstream can then be stored and / or transmitted.

[0055] As mentioned above, some cameras may be suitable for interchangeable lenses. Different lenses can have different characteristics, such as different focal lengths, to meet certain requirements of different use cases. Figure 3b A schematic diagram of the sensor head 320 and the first interchangeable lens 321 of the imaging system 300 is shown. The sensor head 320 includes an image sensor 301.

[0056] Figure 3c A schematic diagram of the imaging system 300 when equipped with the second interchangeable lens 322 is shown.

[0057] Now refer to Figure 4 , Figure 5a , Figure 5b , Figure 5c and Figure 5d And further reference Figure 2a , 2b Exemplary embodiments of lens distortion correction for determining an interchangeable camera lens suitable for use in an interchangeable lens camera are described in 3a, 3b, and 3c. The camera will be... Figure 1b , 2a Take the video camera 120 in 2b as an example. As mentioned above, camera 120 can access some lens distortion correction algorithms, but not all. This can cause problems if the camera is equipped with a lens for which the correct lens distortion correction algorithm cannot be used. For example, camera 120 can access the first lens distortion correction algorithm used for the first interchangeable lens 321, but cannot access the second lens distortion correction algorithm used for the second interchangeable lens 322.

[0058] Lens distortion correction can be a one-dimensional function. Lens distortion correction can be a function based on the distance to the optical center. If the image sensor 301 and lenses 321, 322 are not perfectly aligned, or if the lenses have an unusual shape, then lens distortion correction can be a two-dimensional function. This function can be described in polynomial or tabular form. In some embodiments herein, the first table may, for example, include uncorrected distances and corresponding corrected distances. Linearization can be used between the table values. In some other embodiments herein, the second table may, for example, include a scaling factor representing a linear scaling between the uncorrected distance and the corresponding corrected distance.

[0059] Figure 4Lens distortion correction for the first lens 321 is shown. Specifically, Figure 4 A first uncorrected image frame 421-1 captured by camera 120 using a first lens 321 is shown. The first uncorrected image frame 421-1 needs to be corrected due to lens distortion associated with the first lens 321. The first uncorrected image frame 421-1 includes a first feature 431 at a first uncorrected location 431-1. A feature is a piece of information related to solving a computational task relevant to a specific application. More specifically, in embodiments herein, a feature may be part of an object, and its feature vector is a pattern that identifies it. A feature descriptor may include a feature vector and the location of the feature. The feature may be, for example, an edge, structure, color, color variation, lighting conditions, or light variations.

[0060] In the embodiments described herein, this application aims to find a lens distortion correction method that corrects distorted images by warping pixel data from uncorrected pixel locations to corrected pixel locations. The lens distortion correction method can be found by matching features in the uncorrected and corrected images.

[0061] Feature matching is the task of establishing a correspondence between two features of the same scene / object. Common methods of feature matching involve detecting a set of feature points, each associated with a feature descriptor in image data. Once features or feature points and their descriptors have been extracted from, for example, two or more images, the next step may be to establish some preliminary feature matching between these images.

[0062] Features can be parts or patterns in an image that help identify it as an object. For example, a square has four corners and four sides; these can be called features of a square, and they help identify it as a square. Features include properties such as corners, edges, feature point regions, ridges, etc. Features can also be specific structures in an image, such as points, edges, or objects. These features can be divided into two main categories: A) Features located at specific locations in the image, such as mountain peaks, building corners, doorways, or interestingly shaped snow blocks. These types of localized features are often called keypoint features (even corners) and are typically described by the appearance of the pixel patch surrounding the point location; and B) Features that can be matched based on their orientation and local appearance (edge ​​contours) are called edges, and they can also be good indicators of object boundaries and occlusion events in an image sequence.

[0063] Traditional computer vision techniques for feature detection and feature matching include: Harris angle detection, Shi-Tomasi angle detector, Scale Invariant Feature Transform (SIFT), Accelerated Robust Features (SURF), and Accelerated Segmented Test Features (FAST).

[0064] Traditional feature extractors can be replaced by convolutional neural networks (CNNs) at least in some cases because CNNs have the powerful ability to extract more detailed and complex features that represent images, learn task-specific features, and are more efficient.

[0065] The components for feature detection and matching include:

[0066] 1) Detection: Identifying feature points;

[0067] 2) Description: The local appearance around each feature point can be described in some way, at least ideally, remaining invariant to changes in lighting, transfer, scaling, and in-plane rotation. A descriptor vector for each feature point can be obtained. The appearance can, for example, include the brightness value of each pixel surrounding the feature point.

[0068] 3) Matching: Comparing descriptors between images to identify similar features. For two images, a set of pairs can be obtained. , where (Xi,Yi) is a feature in one image, and (Xi`,Yi`) is its matching feature in another image.

[0069] A feature descriptor is an algorithm that acquires an image and outputs a feature descriptor. A feature descriptor encodes information of interest into a series of numbers and acts as a digital "fingerprint" that can be used to distinguish one feature from another. Ideally, this information is invariant to image transformations, so that the feature can be found again even if the image is transformed in some way. After detecting feature points, a descriptor can be computed for each feature point. Descriptors can be divided into two categories: a) local descriptors and b) global descriptors. A local descriptor is a compact representation of the point's local neighborhood. Local descriptors attempt to resemble the shape and appearance only in the local neighborhood around the point, making them well-suited for representing it in terms of matching. A global descriptor describes the entire image.

[0070] Return to Figure 4 The first uncorrected position 431-1 can be described by coordinates in a coordinate system. In some embodiments herein, the first uncorrected position 431-1 is described by the distance to the center 425-1 of the first uncorrected image frame 421-1. The first uncorrected position 431-1 can also be described by a direction toward the center 425-1. The first uncorrected image frame 421-1 may also include additional features at other locations, such as additional features 441, 451 of the first uncorrected image frame 421-1. The center 425-1 may be an optical center.

[0071] Figure 4A first corrected image frame 421-2, calculated from a first uncorrected image frame 421-1 by means of first lens distortion correction, is also shown. The first corrected image frame 421-1 also includes a first feature 431 at a first corrected position 431-2. The first corrected position 431-2 differs from the first uncorrected position 431-1. The first corrected image frame 421-2 may also include additional features at other corrected positions.

[0072] Figure 4 The center 425-2 of the first corrected image frame 421-2 is also shown. Figure 4 The diagram further illustrates how the first lens distortion correction moves the first feature 431 from the first uncorrected position 431-1 to the first corrected position 431-2. That is, the first uncorrected position 431-1 of the first feature is corrected to the first corrected position 431-2. The first lens distortion correction can also move other features 441 and 451. Correction can be performed relative to the centers 425-1 and 425-2 of the first uncorrected and corrected image frames 421-1 and 421-2.

[0073] For example, while radial distortion is primarily dominated by low-order radial components, it can be corrected using the Brownian distortion model. As mentioned above, the most common distortions are radially symmetric. These radial distortions can be classified, for example, as barrel distortion or pincushion distortion or a mixture of both. Mathematically, barrel distortion and pincushion distortion can be described as quadratic distortions, meaning they increase with the square of the distance from the center.

[0074] As mentioned above, lens distortion correction algorithms can be implemented by warping a distorted input image into a rectified, undistorted output image. This can be achieved by performing an inverse transform; for example, for each pixel (u, v) in the rectified output image, the corresponding coordinates (u, v) in the input image are calculated. ∨ v ∨ ).

[0075] The following calculations are based on a single feature, but can be extended to multiple features, as will be described below.

[0076] The location of a feature can be described by the radius from the center of the corresponding image frame.

[0077] Let r(1, u) be the first uncorrected radius of the feature (e.g., the first feature 431) captured by the first lens 321.

[0078] Let r(1, c) be the first correction radius of the feature captured by the first lens 321 (e.g., the first feature 431) (e.g., after the first uncorrected image frame 421-1 has been corrected).

[0079] The relationship between r(1,c) and r(1,u) can be obtained from calibrations performed, for example, during lens or camera manufacturing.

[0080] For a single feature point, that is, for a single pixel, this is:

[0081] r(1,u)=pr(1,c), where p is a real positive number.

[0082] This relationship is "backwards," from the corrected image to the uncorrected image. This is because the corrected image is actually filled using the pixel values ​​of the uncorrected image. Starting with a "blank" corrected image and model, answer the following question: Which pixels from the uncorrected image should be selected and used to fill that coordinate in the corrected image?

[0083] The best model is probably the one that minimizes the sum:

[0084] |r'(1,u)-r(1,u)|^2, where r is the measured radius and r' is the radius given by the model.

[0085] Finally, we find a p that minimizes |r'(1,u)-r(1,u)|^2=|pr(1,c)-r(1,u)|^2.

[0086] In the case of multiple locations with corresponding features or feature points, the objective may be to minimize the sum of all differences.

[0087] For example, if the function used is a first-degree polynomial:

[0088] r(1,u)=a+b*r(1,c)

[0089] The task here is to select a and b that are the same for all feature points, such that the sum of the difference / error is as low as possible.

[0090] When written as a matrix, it can be viewed as an equation:

[0091] AX≈Y

[0092] Among these, the error |AX-Y| should be minimized as much as possible.

[0093] Y is the actual measured value, A describes the lens distortion correction, i.e., the model, and the X value should be optimized.

[0094] From the example above, r(1,c)[0] is the radius of the 0:te feature, and so on:

[0095] A = [

[0096] 1,r(1,c)[0],

[0097] 1,r(1,c)[1], ... ]

[0100] X = [

[0101] a,

[0102] b, ]

[0104] Y = [

[0105] r(1,u)[0],

[0106] r(1,u)[1], ... ]

[0109] Figure 5a A flowchart is shown describing a method performed in image processing devices 120, 260 for determining lens distortion correction for an interchangeable camera lens (such as a second interchangeable camera lens 322) used by a camera 120 adapted for interchangeable lenses 321, 322.

[0110] In addition, the method can also be used to determine corrections for aberrations caused by different alignments between the first lens 321 and the image sensor 301 and between the second lens 322 and the image sensor 301.

[0111] Image processing devices 120 and 260 may be, for example, video camera 120 (e.g., surveillance camera) or video server 260.

[0112] The following actions can be taken in any suitable order (e.g., in a different order than that presented below).

[0113] Action 501

[0114] The method may include capturing a first uncorrected image frame 421-1 using a first lens 321. The first uncorrected image frame 421-1 captures a first scene.

[0115] Action 502

[0116] The method may further include correcting the first uncorrected image frame 421-1 based on a first lens distortion correction, or correcting detected features (e.g., first feature 431) of the first uncorrected image frame 421-1, or both. Correcting the first feature 431 may include correcting the position of the first feature 431. For example, the first uncorrected position 431-1 may be corrected to a first corrected position 431-2. The first corrected position 431-2 is a first lens distortion correction based on the first uncorrected position 431-1 of the first feature 431.

[0117] The proposed method does not require correction of the entire first uncorrected image frame 421-1. Instead, for embodiments herein, one or more feature descriptors, including a description of the corrected features (including the corrected first feature 431) and their locations, can be acquired and stored.

[0118] Action 503

[0119] The method includes obtaining an indication of a first feature 431 and a first correction position 431-2 of the first feature 431 in the first corrected image frame 421-2. As described above, the first correction position 431-2 is a first lens distortion correction based on the first uncorrected position 431-1 of the first feature 431.

[0120] This method can be applied to multiple features, such as a first feature 431 and another first feature 441. In other words, the first feature 431 may include a set of multiple first features 431, 441.

[0121] Figure 5b Some embodiments are shown for obtaining the first feature 431 and thus the indication of obtaining the first feature 431. The first feature 431 may be obtained from the first background image frame 421b, for example, based on the first corrected image frame 421-2 or based on the first uncorrected image frame 421-1. In detail, Figure 5b A first image frame 421 captured using a first interchangeable lens 321 is shown. The first image frame 421 includes a scene with different objects, which can be classified as background and foreground objects according to some criteria. A first background image frame 421b can be generated, and the first background image frame 421b can include the background objects of the first image frame 421. Correspondingly, a first foreground image frame 421a can be generated, and the first foreground image frame 421a can include the foreground objects of the first image frame 421. Since the first feature 431 is obtained from the first background image frame 421b, the corrected position of the first feature 431 in the image frame captured by the camera 120 should not change due to changes in lenses 321, 322. Therefore, the first corrected position 431-1 obtained from the first background image frame 421b is a good candidate for use as a basis fact.

[0122] Figure 4 A closer look at the first feature 431 is also shown. In some embodiments herein, the first feature 431 includes... Figure 4 Multiple first pixels 431-11 and 431-12 are visible in the zoomed-in portion.

[0123] The first feature 431 may also include multiple first feature points 431-A, 431-B at multiple first pixel locations, which is in Figure 4 It can also be seen in the zoomed-in portion.

[0124] In some embodiments herein, actions 502 and 503 are performed after action 504 or after actions 504 and 505. In some other embodiments herein, actions 502 and 503 are performed even after action 506 (e.g., after actions 504, 505, and 506). In some other embodiments, action 503 includes acquiring a first feature 431 from a first uncorrected image frame 421-1. Then, action 502 may be performed before action 509, such as between actions 507 and 509.

[0125] Action 504

[0126] The method also includes detecting changes in the lens on camera 120 from a first lens 321 to a second interchangeable lens 322.

[0127] The detection of changes in the lens on camera 120 from the first lens 321 to the second lens 322 may include any one or more of the following:

[0128] The characteristics of feature 431 of image frame 421-1 captured by camera 120 when equipped with the first lens 321 are compared with the characteristics of feature 432 of image frame 422-1 captured by camera 120 when equipped with the second lens 322. For example, lens changes can be detected by comparing the characteristics of the first feature 431 with the characteristics of the second feature 432. In some example embodiments, the compared characteristics include feature vectors. In some other example embodiments, the compared characteristics include the position of the features. For example, the pixel positions of features captured by camera 120 when equipped with the first lens 321 can be compared with the pixel positions of features captured by camera 120 when equipped with the second lens 322, and changes in lenses 321, 322 can be detected based on detecting differences in these pixel positions. Furthermore, changes in lenses 321, 322 can be detected based on the distance between these pixel positions. For example, if the distance between pixel positions is greater than a threshold distance, a lens change can be detected. Comparisons can be performed for multiple features. For example, if the distance between pixel positions is greater than a threshold distance for a threshold number of features, a lens change is detected.

[0129] - A first background image captured by camera 120 when equipped with the first lens 321 is compared with a second background image captured by camera 120 when equipped with the second lens 322. For example, a change in lens position can be detected by detecting changes in the pixel positions of matching features. Good candidate features for detecting changes in lens position are, for example, features that are static as long as the lens position remains unchanged.

[0130] - Detect the cessation of the first signal from the first lens 321.

[0131] - Detect a second signal from gyroscope 331, which is correlated with changes in lenses 321 and 322.

[0132] Action 505

[0133] Figure 5c Corresponding to Figure 4 Furthermore, lens distortion correction for the second lens 322 is shown, corresponding to the lens distortion correction shown for the first lens 321. Figure 4 . Specifically, Figure 5c A second uncorrected image frame 422-1 captured by a camera 120 having a second lens 322 is shown. Due to lens distortion associated with the second lens 322, the second uncorrected image frame 422-1 needs to be corrected. The second uncorrected image frame 422-1 includes a second feature 432 at a second uncorrected position 432-1. The second feature 432 may correspond to a first feature 431 captured using the first lens 321. The second feature 432 may be the same feature as the first feature 431. Since the second uncorrected image frame 422-1 was captured by the camera 120 using the second lens 322, the second uncorrected position 432-1 is likely to differ from the first uncorrected position 431-1. When the uncorrected positions of corresponding features differ, the first lens distortion correction will not be applicable to image frames captured using the second lens 322, such as the second uncorrected image frame 422-1. The embodiments herein aim to find a second lens distortion correction that corrects the second uncorrected position 432-1 to the first corrected position 431-2.

[0134] The second uncorrected position 432-1 can be described by coordinates in a coordinate system. In some embodiments herein, the second uncorrected position 432-1 is described by the distance to the center 426-1 of the second uncorrected image frame 422-1. The second uncorrected image frame 422-1 may also include additional features at other locations, such as additional features 442, 452.

[0135] Figure 5cA second corrected image frame 422-2 calculated from a second uncorrected image frame 422-1 by means of second lens distortion correction is also shown. Embodiments herein disclose a method for calculating second lens distortion correction. The second corrected image frame 422-1 further includes a second feature 432 at a second correction position 432-2. The second corrected image frame 422-2 may also include additional features at other positions, such as additional features 442, 452. The purpose of the second lens distortion correction may be to minimize the distance between the first feature 431 at a first correction position 431-2 (e.g., given by pixel position) in the first corrected image frame 421-2 and the corresponding second feature 432 at a second correction position 432-2 (e.g., given by pixel position) in the second corrected image frame 422-2.

[0136] The method may also include capturing a second uncorrected image frame 422-1 using a second lens 322.

[0137] This method also applies when the first field of view (FOV) of the first uncorrected image frame 421-1 is different from the second FOV of the second uncorrected image frame 422-1, or when there is a global offset between the first uncorrected image frame 421-1 and the second uncorrected image frame 422-1, or both. The FOV can be obtained from the sensor size and focal length f. For example, the FOV can be calculated as 2 times arctan(sensor size / 2f).

[0138] Action 506

[0139] The method further includes obtaining an indication of a second feature 432 corresponding to the first feature 431, and a second uncorrected position 432-1 of the second feature 432 in a second uncorrected image frame 422-1 captured by the camera 120 when equipped with the second lens 322. The second uncorrected image frame 422-1 captures a second scene that at least partially overlaps with the first scene.

[0140] Figure 5d This corresponds to the illustration of obtaining the first feature 431 from the background image frame 421b. Figure 5b . Figure 5d Some embodiments are shown that indicate how to obtain the second feature 432 and thus the second feature 432. The second feature 432 can be obtained from the second background image frame 422b based on the second uncorrected image frame 422-1. In detail, Figure 5dA second image frame 422 captured using a second interchangeable lens 322 is shown. The second image frame 422 includes a scene with different objects, which can be categorized into background objects and foreground objects according to some criteria. A second background image frame 422b can be generated, and the second background image frame 422b can include the background objects of the second image frame 422. Correspondingly, a second foreground image frame 422a can be generated, and the second foreground image frame 422a can include the foreground objects of the second image frame 422.

[0141] Figure 5c A zoom-in of the second feature 432 is also shown. In some embodiments herein, the second feature 432 includes a plurality of second pixels 432-11, 432-12.

[0142] The second feature 432 may include multiple second feature points 432-A, 432-B at multiple second pixel locations. These multiple second pixel locations may correspond to multiple first pixel locations.

[0143] This method can be applied to multiple features, such as a second feature 432 and another second feature 442. In other words, the second feature 432 may include a set of multiple second features 432, 442.

[0144] Action 507

[0145] The method further includes matching a first feature 431 with a second feature 432. For matching, the first feature 431 can be derived from a first uncorrected image frame 421-1 or from a first corrected image frame 421-2. Therefore, in some embodiments, the image processing devices 120, 260 match the first feature 431 in the first corrected image frame 421-2 with the second feature 432 in the second uncorrected image frame 422-1. In some other embodiments, the image processing devices 120, 260 match the first feature 431 in the first uncorrected image frame 421-1 with the second feature 432 in the second uncorrected image frame 422-1.

[0146] Matching can include comparing descriptors of features across an image to identify similar features. For example, matching a first feature 431 with a second feature 432 can include comparing a first descriptor of the first feature 431 with a second descriptor of the second feature 432. For instance, a first feature vector of the first feature 431 can be compared with a second feature vector of the second feature 432. This comparison can produce a numerical result that can be evaluated against a matching threshold. For example, if the metric for matching the first feature 431 with the second feature 432 is greater than the matching threshold, then the first feature 431 matches the second feature 432. Matching can be based on a search distance algorithm, such as least squares difference.

[0147] For example, a feature descriptor a = [a1, a2, ...] can be a mathematical vector, which can be compared with the corresponding feature descriptor b = [b1, b2, ...] using the vector norm.

[0148] |ab|=sqrt((a1-b1)^2+(a2-b2)^2…

[0149] Feature matching can be used to find or derive attributes and transfer them from a source image to a target image. Analyzing the feature attributes of an image helps determine the correct match during feature matching. When one or more matching fields are specified by an algorithm or model, spatial matching features can be checked against these fields. If a source feature spatially matches two or more candidate target features, but one target feature has a matching attribute value while the other does not, the detected match can be selected as the final match. Some examples of feature matching methods have been mentioned above. Another example of image feature matching methods is the local feature transformer.

[0150] When a feature includes multiple feature points, matching the first feature 431 with the second feature 432 may include matching each of the multiple first feature points 431-A and 431-B with the corresponding feature point of the multiple second feature points 432-A and 432-B.

[0151] Action 508

[0152] In some embodiments herein, the method further includes selecting a first feature 431 from a first set of features 431, 441, 451, for example, in a first corrected image frame 421-2, and selecting a corresponding second feature 432 from a second set of features 432, 442, 452, for example, in a second uncorrected image frame 422-1, based on the degree to which the first feature 431 matches the second feature 432. Thus, the selected feature can be chosen from the matching features. The degree to which the first feature 431 matches the second feature 432 can be evaluated, for example, based on a matching metric described above in action 507. Such a metric can be, for example, a value between 0 and 1, where 0 indicates a mismatch and 1 indicates a perfect match. The least-squares difference of feature vectors can also be used.

[0153] The method can be executed for other features. For example, the method can be executed for a set of multiple first features 431, 441 and a set of multiple second features 432, 442.

[0154] Then, selecting a plurality of first features 431, 441 from the first set of features 431, 441, 451, and a corresponding plurality of second features 432, 442 from the second set of features 432, 442, 452, can further be based on the overlapping geographic representation of the first and second scenes through the selected plurality of first features 431, 441 and the selected plurality of second features 432, 442. In some embodiments herein, the selection of the corresponding plurality of first and second features maximizes the overlapping geographic representation.

[0155] A geographic representation can be determined based on the corresponding radial distances of the selected features from the geographic centers 425-1, 425-2, 426-1 of the image frames in which the features appear. The geographic centers 425-1, 425-2, 426-1 can be optical centers. For example, a geographic representation can be determined based on the spread of the corresponding radial distances of the selected features from the geographic centers 425-1, 425-2, 426-1 of the image frames in which the features appear. In some embodiments herein, a selected set of multiple first features 431, 441 and a selected set of multiple second features 432, 442 are each selected such that they maximize the spread of the radial distances of the selected features from the geographic centers 425-1, 425-2, 426-1.

[0156] For example, features can be selected from a larger set of features (e.g., all features) based on their radial location, such that the ensemble of selected features covers different radial locations from the minimum to the maximum radial location of the overlap. Therefore, features can be selected based on how well they represent the possible radial locations of the overlap as a whole. In the first example, there might be many features close to the center of the overlap. The geographic coverage could be considered poor. In the second example, the features can be radially spread across the overlap. The geographic coverage could be considered good.

[0157] Therefore, features can be selected based on the degree of geographical overlap represented by the selected features. The degree of geographical overlap represented by the selected features can be measured by the expansion of radial distances to centers 425-1, 425-2. This selection may include calculating the radial distances of multiple features (e.g., all matching features) to centers 425-1, 425-2, and then selecting a set of features from the multiple features, for example, including a fixed number of features, that maximizes the expansion of the distribution of the selected features' locations. For example, the expansion of radial distances of a selected set of multiple features can be calculated for a group of multiple alternative selections and then compared. However, the number of features can also be dynamic, for example, depending on the matching score and / or the degree of radial expansion of the features from the center of the image frame. As mentioned above, the center of the image frame can be the optical center. In some example embodiments, image processing devices 120, 260 select additional feature pairs, including a first feature and a second feature, that have a high matching score (e.g., exceeding a matching score threshold) and are radially furthest from the already selected feature pairs. The location of the feature pair can be calculated as the average location of the first feature and the second feature. In some embodiments of this document, the position of the first candidate feature pair is radially furthest from the center of the image frame when the difference between the radial distance of the first candidate feature pair to the center of the image frame and the corresponding radial distance of the selected feature pair to the center of the image frame is greater than the second corresponding difference between the second radial distance of any other second candidate feature pair to the center of the image frame and the corresponding radial distance of the selected feature pair to the center of the image frame.

[0158] If fewer feature pairs are selected in the overlapping region, the matching score threshold can be lowered. If more feature pairs are selected in the overlapping region, the matching score threshold can be increased.

[0159] Therefore, selecting a set of multiple first features 431, 441 from the first set of features 431, 441, 451 and selecting a corresponding set of multiple second features 432, 442 from the second set of features 432, 442, 452 can be based on the extension of the corresponding radial distance in the selected set of multiple features.

[0160] In other words, selecting a plurality of first features 431, 441 from the first set of features 431, 441, 451 can be based on the expansion of the corresponding radial distances of the selected plurality of first features 431, 441. Correspondingly, selecting a plurality of second features 432, 442 from the second set of features 432, 442, 452 can be based on the expansion of the corresponding radial distances of the selected plurality of second features 432, 442.

[0161] If lens aberrations vary with radial distance to the optical center, then a choice based on radial position may be advantageous.

[0162] In some embodiments herein, the selection of features for mapping may be performed prior to feature matching in action 507 described above. If action 507 is performed after action 508, further feature selection may be based on matching criteria, such as a matching threshold. For example, feature pairs such as first feature 431 and second feature 432 may be selected based on a matching threshold for further processing described below.

[0163] Action 509

[0164] The method further includes determining second lens distortion correction for the second lens 322 based on the mapping between the first corrected position 431-2 of the matched first feature 431 and the second uncorrected position 432-1 of the matched second feature 432. The mapping between the first corrected position 431-2 of the matched first feature 431 and the second uncorrected position 432-1 of the matched second feature 432 can be a mapping in any direction. For example, the mapping between the first corrected position 431-2 of the matched first feature 431 and the second uncorrected position 432-1 of the matched second feature 432 can be a mapping from the second uncorrected position 432-1 of the matched second feature 432 to the first corrected position 431-2 of the matched first feature 431.

[0165] As mentioned above in action 507, in some embodiments, image processing devices 120, 260 match a first feature 431 in a first uncorrected image frame 421-1 with a second feature 432 in a second uncorrected image frame 422-1. Then, determining the second lens distortion correction for the second lens 322 can be based on the mapping between the first uncorrected position 431-1 of the matched first feature 431 and the second uncorrected position 432-1 of the matched second feature 432, combined with the first lens distortion correction of the first uncorrected position 431-1 of the first feature 431 in the first uncorrected image frame 421-1. Finally, this will also result in the aforementioned mapping from the second uncorrected position 432-1 of the matched second feature 432 to the first corrected position 431-2 of the matched first feature 431, because, as described above, the first lens distortion correction of the first uncorrected position 431-1 of the first feature 431 moves the first feature 431 from the first uncorrected position 431-1 to the first corrected position 431-2. That is, the first uncorrected position 431-1 of the first feature is corrected to the first corrected position 431-2.

[0166] Therefore, the mapping between the first corrected position 431-2 of the matched first feature 431 and the second uncorrected position 432-1 of the matched second feature 432 can be obtained by combining the mapping between the first uncorrected position 431-1 of the matched first feature 431 and the second uncorrected position 432-1 of the matched second feature 432 with the mapping between the first corrected position 431-2 of the matched first feature 431 and the first uncorrected position 431-1.

[0167] The mapping can be determined based on minimizing the distance between the first corrected position 431-2 of the matched first feature 431 and the second corrected position 432-2 of the matched second feature 432. The second corrected position 432-2 is calculated based on applying the second lens distortion correction to the second uncorrected position 432-1.

[0168] The distance between the first corrected position 431-2 of the matched first feature 431 and the second corrected position 432-2 of the matched second feature 432 can be minimized by using the least squares method. For example, the position of a feature can be described by the radius to the center of the corresponding image frame.

[0169] Let r(1, c) be the first correction radius of the feature of the first lens 321 (e.g., the first feature 431) (e.g., after the first uncorrected image frame 421-1 has been corrected).

[0170] Let r(2, u) be the uncorrected radius of the same or corresponding feature (e.g., second feature 432) captured by the second lens 322.

[0171] Let r(2, c) be the correction radius of the same or corresponding feature (e.g., second feature 432) captured by the second lens 322 (e.g., after the second uncorrected image 422-1 has been corrected).

[0172] The question then becomes how to select pixels from the second uncorrected image frame 422-1 captured with the second lens 322 and ensure it resembles the first corrected image frame 421-2 captured with the first lens 321. In other words, it can be assumed that r(1,c) = r(2,c), meaning that the embodiments described herein aim to utilize the second lens 322 to obtain the same geometry as the first lens 321. In practice, a constant factor corresponding to digital zoom-in or zoom-out bits can be added, which does not change the discussion. For example, such a constant factor might be necessary if the two lenses capture the scene with very different FoVs.

[0173] Then let r(2,u) = qr(1,c), where q is the scaling factor, i.e., a positive real number.

[0174] |r'(2,u)-r(2,u)|^2, where r is the measured radius and r' is the radius given by the model, such as for second-lens distortion correction.

[0175] The second lens distortion correction can be found by finding q that minimizes |r'(1,c)-r(2,u)|^2=|qr(1,c)-r(2,u)|^2.

[0176] In the first embodiment, r(1, c) is obtained directly from the first corrected image frame 421-2 of the first feature 431, while r(2, u) is obtained by direct measurement in the second uncorrected image frame 422-1 of the second feature 432.

[0177] The second embodiment is similar, but here r(1, c) is instead obtained directly from the first uncorrected image frame 421-1 of the first feature 431, giving r(1, u). Then, the calibration from the manufacturing process can be applied to the first uncorrected position 431-1 of the first feature 431 to obtain r(1, c). In this way, it is not necessary to perform actual correction on the first uncorrected image frame 421-1 captured using the first lens 321.

[0178] It is necessary to reverse the first-lens distortion correction, i.e., to give a calibration model where r(1,u) < -r(1,c). This is possible in most cases of interest.

[0179] If the correction is described by the parameters of a polynomial, the mapping may include the parameters of a polynomial that minimize the distance between the first correction position 431-2 of the matched first feature 431 and the second correction position 432-2 of the matched second feature 432.

[0180] The embodiments described herein can be performed for features comprising multiple pixels. Mapping can then be performed on each of the plurality of first pixels 431-11, 431-12 and each corresponding pixel of the plurality of second pixels 432-11, 432-12.

[0181] When the first feature 431 includes a set of multiple first features 431, 441 and the second feature 432 includes a corresponding set of multiple second features 432, 442, the embodiments of this invention can be performed, and specifically, action 509 is performed. Then, the embodiments of this invention can be performed on multiple pairs of first and second features 431, 432, and specifically, action 509 is performed. Each of the multiple features in this set can include multiple pixels, and mapping can be performed on corresponding pixel pairs of each pair of corresponding features.

[0182] In some embodiments, the second lens distortion correction is determined such that the corrected image frame 422-2 captured by the camera 120 when equipped with the second lens 322 is less affected by lens distortion than the uncorrected image frame 422-1 captured by the camera 120 when equipped with the second lens 322. For example, the second lens distortion correction may be determined such that the second corrected image frame 422-2 is less affected by lens distortion than the second uncorrected image frame 422-1.

[0183] Action 510

[0184] The method may also include applying second lens distortion correction to uncorrected image frames 422-1 captured by camera 120 when equipped with second lens 322.

[0185] Figure 5c The center 426-2 of the second corrected image frame 422-2 is also shown. Figure 5c The diagram also illustrates how the second lens distortion correction moves the second feature 432 from the second uncorrected position 432-1 to the second corrected position 432-2. That is, the second uncorrected position 432-1 of the second feature is corrected to the second corrected position 432-2. The second lens distortion correction can also move other features 442 and 452. Correction can be performed relative to the centers 426-1 and 426-2 of the second uncorrected and corrected image frames 422-1 and 422-2.

[0186] As described above, the first feature 431 can be obtained from the first background image 421b based on the first corrected image frame 421-2, and the second feature 432 can be obtained from the second background image 422b based on the second uncorrected image frame 422-1.

[0187] In some embodiments of this document, the first feature 431 includes a plurality of first pixels 431-11 and 431-12, and the second feature 432 includes a plurality of second pixels 432-11 and 432-12.

[0188] The first feature 431 may include multiple first feature points 431-A and 431-B at multiple first pixel locations, and the second feature 432 may include multiple second feature points 432-A and 432-B at multiple second pixel locations. The multiple second pixel locations may correspond to multiple first pixel locations.

[0189] In the embodiments described herein, an advantage is that the first and second features 431, 432 do not need to be associated with a straight line or a known reference mark. Because the first and second features 431, 432 do not need to be associated with a straight line or a known reference mark, the requirements for performing this method are more relaxed compared to other methods.

[0190] Therefore, in some embodiments herein, the first and second features 431, 432 are not associated with a straight line or with a known reference mark.

[0191] refer to Figure 6 A schematic block diagram of an embodiment of the image processing device 600 is shown. The image processing device 600 corresponds to... Figure 1b The image processing device 110. Therefore, the image processing device 600 may include any one of a camera (e.g., a surveillance camera, monitoring camera, video camera, network video recorder) and a wireless communication device 130. Specifically, the image processing device 600 may be a camera 120 (e.g., a surveillance video camera) or a video server 260.

[0192] As described above, the image processing device 600 is configured to determine lens distortion correction for the interchangeable camera lens (e.g., the second interchangeable camera lens 322) used by the camera 120 suitable for interchangeable lenses 321, 322.

[0193] The image processing apparatus 600 may also include a processing module 601, such as means for performing the methods described herein. This means may be implemented as one or more hardware modules and / or one or more software modules.

[0194] The image processing apparatus 600 may also include a memory 602. The memory may include instructions, such as including or storing instructions, for example in the form of a computer program 603, which may include computer-readable code units that, when executed on the image processing apparatus 600, cause the image processing apparatus 600 to perform a method for determining lens distortion correction for an interchangeable camera lens.

[0195] Image processing device 600 may include a computer, on which computer-readable code units may be executed and cause the computer to perform a method for determining a probability value indicating that an object captured in an image frame stream belongs to that object type.

[0196] According to some embodiments herein, the image processing device 600 and / or processing module 601 includes a processing circuit 604 as an exemplary hardware module, which may include one or more processors. Therefore, the processing module 601 may be embodied in, or "implemented" by, the processing circuit 604. Instructions may be executed by the processing circuit 604, thereby enabling the image processing device 600 to perform as described above. Figure 5a The method. As another example, when executed by the image processing device 600 and / or the processing circuit 604, the instructions can cause the image processing device 600 to perform according to... Figure 5a The method.

[0197] In view of the above, in one example, an image processing device 600 is provided for determining lens distortion correction for interchangeable camera lenses.

[0198] Similarly, memory 602 contains instructions executable by the processing circuitry 604, thereby enabling image processing device 600 to perform operations according to... Figure 5a The method.

[0199] Figure 6 A carrier 605 or program carrier is also shown, which includes a computer program 603 as directly described above. The carrier 605 may be one of an electrical signal, an optical signal, a radio signal, or a computer-readable medium.

[0200] In some embodiments, the image processing device 600 and / or processing module 601 may include one or more of an acquisition module 610, a detection module 620, a matching module 630, a selection module 640, a mapping module 650, and a correction module 660, as an example of a hardware module. In other embodiments, one or more of the foregoing exemplary hardware modules may be implemented as one or more software modules.

[0201] Furthermore, the processing module 601 may include an input / output unit 606. According to an embodiment, the input / output unit 606 may include an image sensor configured to capture the aforementioned raw image frames, such as raw image frames included in the video stream 310 from the image sensor 301.

[0202] According to the various embodiments described above, the image processing device 600 and / or the processing module 601 and / or the acquisition module 610 are configured to acquire an indication of the first feature 431 and a first corrected position 431-2 of the first feature 431 in a first corrected image frame 421-2. The first corrected position 431-2 is based on a first lens distortion correction that includes the first uncorrected position 431-1 of the first feature 431 in a first uncorrected image frame 421-1, which captures the first scene.

[0203] The image processing device 600 and / or processing module 601 and / or acquisition module 610 are further configured to acquire an indication of a second feature 432 corresponding to the first feature 431 and a second uncorrected position 432-1 of the second feature 432 in a second uncorrected image frame 422-1 captured by the camera 120 when equipped with the second lens 322, the second uncorrected image frame 422-1 capturing a second scene that at least partially overlaps with the first scene.

[0204] In some embodiments herein, the first feature 431 is obtained from the first background image frame 421b, and the second feature 432 is obtained from the second background image frame 422b.

[0205] Image processing device 600 and / or processing module 601 and / or detection module 620 are configured to detect changes in the lens on camera 120 from first lens 321 to second interchangeable lens 322.

[0206] The image processing device 600 and / or processing module 601 and / or matching module 630 are further configured to match the first feature 431 with the second feature 432.

[0207] The image processing device 600 and / or processing module 601 and / or determination module 640 are further configured to determine the second lens distortion correction of the second lens 322 based on the mapping between the first corrected position 431-2 of the matched first feature 431 and the second uncorrected position 432-1 of the matched second feature 432.

[0208] In some embodiments herein, the image processing device 600 and / or processing module 601 and / or determination module 640 are further configured to determine a second lens distortion correction such that the corrected image frame 422-2 is subject to less lens distortion than the uncorrected image frame 422-1 captured by the camera 120 when equipped with the second lens 322.

[0209] The image processing device 600 and / or processing module 601 and / or correction module 670 may also be configured to apply second lens distortion correction to uncorrected image frames 422-1 captured by camera 120 when equipped with second lens 322.

[0210] In some embodiments herein, image processing device 600 and / or processing module 601 and / or detection module 620 are configured to detect a change in the lens on camera 120 from first lens 321 to second lens 322 by being configured to perform one or more of the following:

[0211] -The characteristics of feature 431 of image frame 421-1 captured by camera 120 when equipped with first lens 321 are compared with the characteristics of feature 432 of image frame 422-1 captured by camera 120 when equipped with second lens 322;

[0212] - Detect the cessation of the first signal from the first lens 321; and

[0213] - Detect a second signal from gyroscope 331, which is correlated with changes in lenses 321 and 322.

[0214] The image processing device 600 and / or processing module 601 and / or determination module 640 can also be configured to determine the mapping based on minimizing the distance between the first correction position 431-2 and the second correction position 432-2.

[0215] The image processing device 600 and / or processing module 601 and / or determination module 640 may also be configured to calculate the second corrected position 432-2 based on applying the second lens distortion correction to the second uncorrected position 432-1.

[0216] The image processing device 600 and / or processing module 601 and / or selection module 650 may also be configured to select the first feature 431 from the first set of features 431, 441, 451 and the corresponding second feature 432 from the second set of features 432, 442, 452 based on the degree of matching between the first feature 431 and the second feature 432.

[0217] When the method is performed for multiple first features 431, 441 and multiple second features 432, 442, the image processing device 600 and / or processing module 601 and / or selection module 650 may also be configured to further select multiple first features 431, 441 from the first set of features 431, 441, 451 and select multiple corresponding second features 432, 442 from the second set of features 432, 442, 452 based on the radial distance of the geographic centers 425-1, 425-2, 426-1, 426-2 of the image frames appearing according to the selected feature distance features, and to select multiple first features 431, 441 and multiple second features 432, 442 from the second set of features 432, 442, 452.

[0218] As used herein, the term "module" can refer to one or more functional modules, each of which can be implemented as one or more hardware modules and / or one or more software modules and / or combinations of software / hardware modules. In some examples, a module can represent a functional unit implemented as software and / or hardware.

[0219] As used herein, the terms "computer program carrier," "program carrier," or "carrier" can refer to one of the following: electronic signal, optical signal, radio signal, and computer-readable medium. In some examples, a computer program carrier may exclude transient, propagating signals, such as electronic, optical, and / or radio signals. Therefore, in these examples, a computer program carrier can be a non-transitory carrier, such as a non-transitory computer-readable medium.

[0220] As used herein, the term "processing module" can include one or more hardware modules, one or more software modules, or a combination thereof. Any such module, whether hardware, software, or a combination of hardware and software modules, can be a connection device, providing device, configuring device, responding device, disabling device, etc., as disclosed herein. By way of example, the term "device" can refer to a module corresponding to the modules listed above in conjunction with the accompanying drawings.

[0221] As used in this article, the term "software module" can refer to software applications, dynamic link libraries (DLLs), software components, software objects, objects based on the Component Object Model (COM), software functions, software engines, executable binary software files, etc.

[0222] The terms "processing module" or "processing circuitry" may encompass processing units, including, for example, one or more processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. Processing circuitry may include one or more processor cores.

[0223] As used herein, the expression “configured to / for” can mean that the processing circuitry is configured (e.g., adapted to or operated as) to perform one or more of the actions described herein through software and / or hardware configuration.

[0224] As used herein, the term "action" can refer to a movement, step, operation, response, reaction, activity, etc. It should be noted that, where applicable, an action in this document can be divided into two or more sub-actions. Furthermore, it should also be noted that two or more actions described herein can be combined into a single action.

[0225] As used herein, the term "memory" can refer to hard disks, magnetic storage media, portable computer floppy disks or floppy disks, flash memory, random access memory (RAM), etc. Additionally, the term "memory" can refer to the processor's internal register memory, etc.

[0226] As used herein, the term "computer-readable medium" can mean a Universal Serial Bus (USB) memory, a DVD, a Blu-ray disc, a software module that receives data as a stream, flash memory, a hard disk drive, a memory card (such as a Memory Stick, a Multimedia Card (MMC), a Secure Digital Card (SD), etc.). One or more of the foregoing examples of computer-readable media can be provided as one or more computer program products.

[0227] As used herein, the term “computer-readable code unit” can be the text of a computer program, a portion or the entire binary file representing a compiled format of a computer program, or anything in between.

[0228] As used herein, the terms “number” and / or “value” can be any kind of number, such as binary, real, imaginary, or rational numbers. Furthermore, “number” and / or “value” can be one or more characters, such as letters or a string of letters. “Number” and / or “value” can also be represented by a bit string, i.e., multiple zeros and / or multiple ones.

[0229] As used herein, the phrase "in some embodiments" has been used to indicate that features of the described embodiments may be combined with any other embodiments disclosed herein.

[0230] Although embodiments of various aspects have been described, many different variations, modifications, etc., will become apparent to those skilled in the art. Therefore, the described embodiments are not intended to limit the scope of this disclosure.

Claims

1. A method executed in an image processing apparatus (120, 260, 600) for determining lens distortion correction for an interchangeable camera lens suitable for use with an interchangeable lens camera, the method comprising: - Obtain an indication of a first feature (431) and a first corrected position (431-2) of the first feature (431) in a first corrected image frame (421-2), the first corrected position (431-2) being calculated based on a first lens distortion correction of the first uncorrected position (431-1) of the first feature (431) in a first uncorrected image frame (421-1), the first uncorrected image frame (421-1) including the first feature (431) and being captured by the camera when equipped with a first interchangeable lens, the first uncorrected image frame (421-1) capturing a first scene; - Detect the change of the lens on the camera (504) from the first interchangeable lens to the second interchangeable lens; - Obtain an indication of a second feature (432) corresponding to the first feature (431) and a second uncorrected position (432-1) of the second feature (432) in a second uncorrected image frame captured by the camera when equipped with the second interchangeable lens, the second uncorrected image frame capturing a second scene that at least partially overlaps with the first scene; - Match the first feature (431) with the second feature (432) (507); and - Based on the mapping between the first corrected position (431-2) of the matched first feature (431) and the second uncorrected position (432-1) of the matched second feature (432), determine (509) the second lens distortion correction of the second interchangeable lens.

2. The method according to claim 1, wherein, The mapping between the first corrected position (431-2) of the matched first feature (431) and the second uncorrected position (432-1) of the matched second feature (432) is based on minimizing the distance between the first corrected position (431-2) and the second corrected position (432-2), which is calculated based on applying the second lens distortion correction to the second uncorrected position (432-1).

3. The method according to claim 1 or 2, wherein, The first feature (431) is obtained from the first background image frame (421b), and the second feature (432) is obtained from the second background image frame (422b) based on the second uncorrected image frame.

4. The method according to any one of claims 1-2, wherein, The first feature (431) includes a plurality of first pixels, and the second feature (432) includes a plurality of second pixels.

5. The method according to any one of claims 1-2, wherein, The first and second features are not associated with a straight line, nor with any known reference markings.

6. The method according to any one of claims 1-2, further comprising: The second lens distortion correction (510) is applied to the uncorrected image frames captured by the camera when it is equipped with the second interchangeable lens.

7. The method according to claim 6, wherein, The second lens distortion correction is determined such that the corrected image frame (422-2) is less affected by lens distortion than the uncorrected image frame captured by the camera when equipped with the second interchangeable lens.

8. The method according to any one of claims 1-2, wherein, Detecting the change of the lens on the camera from the first interchangeable lens to the second interchangeable lens includes any one or more of the following: - The characteristics of the features of the image frames captured by the camera when it is equipped with the first interchangeable lens are compared with the characteristics of the features of the image frames captured by the camera when it is equipped with the second interchangeable lens; - Detecting the cessation of the first signal, the first signal originating from the first interchangeable lens; and - Detect a second signal from the gyroscope (331), which is correlated with changes in the lens.

9. The method according to any one of claims 1-2, further comprising: Select (508) the first feature (431) from the first set of features, and select (508) the corresponding second feature (432) from the second set of features based on the degree of matching between the first feature (431) and the second feature (432).

10. The method according to any one of claims 1-2, wherein, The first feature (431) is included in a set of multiple first features, the second feature (432) is included in a corresponding set of multiple second features, and the method is performed for multiple pairs of first features and second features.

11. The method according to claim 10, wherein, The selection of (508) a set of multiple first features from the first set of features and the selection of (508) a set of multiple second features from the second set of features are also based on the geographic representation of the overlap between the first scene and the second scene by the selected set of multiple first features and the set of multiple second features.

12. An image processing apparatus (120, 260, 600) comprising a processing module (601) and a memory (602); the memory (602) comprising computer-readable code units that, when executed on the image processing apparatus, cause the image processing apparatus to perform the method according to any one of claims 1-11.

13. The image processing apparatus (120, 260, 600) according to claim 12, wherein, The image processing device (120, 260, 600) is the camera or the video server.

14. A computer-readable medium (605) having thereon stored a computer program (603) including computer-readable code units, which, when executed on an image processing device (120, 260, 600), cause the image processing device (120, 260, 600) to perform the method according to any one of claims 1-11.

Images