Image processing device

Abstract

An embodiment of the present invention discloses an image processing apparatus including: a feature point extractor that detects a plurality of feature points from a first image input from an image sensor; a first motion vector extractor that extracts local motion vectors of the plurality of feature points, and selects valid local motion vectors among the extracted local motion vectors according to a zoom ratio of the image sensor; a second motion vector extractor that extracts a global motion vector using the selected valid local motion vectors; and an image stabilizer that corrects a shake of the first image based on the global motion vector.

Description

Technical Field

[0001] This embodiment relates to an image processing apparatus and an image processing method. Background Technology

[0002] Recently, with the increasing use of multimedia devices, the demand for image enhancement technologies for digital images captured in various environments has also increased. Image enhancement technologies include image blur removal, noise reduction, and image stabilization, and are widely used in shooting devices such as digital cameras, smartphones, home cameras or camcorders, industrial surveillance cameras, broadcast cameras, and military imaging devices. Summary of the Invention

[0003] The problem the invention aims to solve

[0004] The present invention provides a method and apparatus for outputting stable images with shake removed, even when an external impact occurs on the image capturing device or when there is a moving object in the captured image.

[0005] means for solving problems

[0006] An image processing apparatus according to an embodiment of the present invention includes: a feature point extractor that detects a plurality of feature points from a first image input from an image sensor; a first motion vector extractor that extracts local motion vectors of the plurality of feature points and uses different algorithms according to the zoom ratio of the image sensor to select effective local motion vectors from the extracted local motion vectors; a second motion vector extractor that extracts global motion vectors using the selected effective local motion vectors; and an image stabilizer that corrects jitter in the first image based on the global motion vectors.

[0007] When the first image is an image captured at high magnification, the first motion vector extractor can utilize the motion region detected in the second image input from the image sensor prior to the first image to select effective local motion vectors from the local motion vectors of the first image.

[0008] When the first image is a high-magnification image, the first motion vector extractor can set a first target region smaller than the foreground region in the first image corresponding to the motion region detected in the second image, and calculate the average size of the local motion vectors existing in the first target region. The first motion vector extractor can also set a second target region larger than the foreground region in the first image, and select local motion vectors existing in the second target region as effective local motion vectors if the difference between the size of the local motion vector and the average size is greater than or equal to a critical value.

[0009] The horizontal and vertical dimensions of the first target region can be twice the maximum value of the local motion vector dimensions in the first image, respectively, compared to the horizontal and vertical dimensions of the foreground region.

[0010] The horizontal and vertical dimensions of the second target region can be twice the maximum value of the local motion vector dimensions in the first image, respectively, compared to the horizontal and vertical dimensions of the foreground region.

[0011] The average size of the local motion vector may include the average size of the x-direction component and the average size of the y-direction component.

[0012] When the first image is an image captured at normal magnification, the first motion vector extractor can use sensor data from the motion sensor to select the effective local motion vector from the local motion vectors of the first image.

[0013] When the first image is an image captured at normal magnification, the first motion vector extractor can generate a sensor vector based on the sensor data, and select the following local motion vectors from the local motion vectors of the first image as the effective local motion vectors: the difference between the size and direction of the local motion vector and the size and direction of the sensor vector is less than or equal to a critical value.

[0014] The image processing apparatus may further include a motion region detector to detect motion regions in the first image after the jitter has been corrected.

[0015] An image processing apparatus according to an embodiment of the present invention includes: a feature point extractor for detecting a plurality of feature points from an image input by an image sensor; a motion vector extractor for extracting local motion vectors of the plurality of feature points; a motion region detector for detecting motion regions from the image; and an image stabilizer for correcting image jitter using the local motion vectors and the motion regions. The motion vector extractor, within a foreground region of the current image corresponding to a motion region detected in a previous image, sets a first target region smaller than the foreground region and calculates an average size of local motion vectors existing within the first target region; and sets a second target region in the current image larger than the foreground region; among the local motion vectors existing within the second target region, removes local motion vectors whose difference between the size of the local motion vector and the average size is less than a threshold value, and selects local motion vectors whose difference is greater than or equal to the threshold value.

[0016] The image can be an image captured by the image sensor at high magnification zoom.

[0017] An image jitter correction method of an image processing apparatus according to an embodiment of the present invention includes: a step of detecting a plurality of feature points from a first image input by an image sensor; a step of extracting local motion vectors of the plurality of feature points and using different algorithms according to the zoom ratio of the image sensor to select effective local motion vectors from the extracted local motion vectors; a step of extracting global motion vectors using the selected effective local motion vectors; and a step of correcting jitter in the first image based on the global motion vectors.

[0018] The step of selecting the effective local motion vector may include the following steps: when the first image is an image captured at a high magnification, using the motion region detected in a second image input from the image sensor prior to the first image, select an effective local motion vector from the local motion vectors of the first image.

[0019] The step of selecting the effective local motion vector may include the following steps: when the first image is an image captured at high magnification, in the foreground region of the first image corresponding to the motion region detected in the second image, a first target region smaller than the foreground region is set, and the average size of the local motion vectors existing in the first target region is calculated; and a second target region larger than the foreground region is set in the first image, and among the local motion vectors existing in the second target region, the following local motion vector is selected as the effective local motion vector: the difference between the size of the local motion vector and the average size is greater than or equal to a critical value.

[0020] The horizontal and vertical dimensions of the first target region can be twice the maximum value of the local motion vector dimensions in the first image, respectively, compared to the horizontal and vertical dimensions of the foreground region.

[0021] The horizontal and vertical dimensions of the second target region can be twice the maximum value of the local motion vector dimensions in the first image, respectively, compared to the horizontal and vertical dimensions of the foreground region.

[0022] The average size of the local motion vector may include the average size of the x-direction component and the average size of the y-direction component.

[0023] The step of selecting the effective local motion vector may include the following steps: when the first image is an image captured at normal magnification, the effective local motion vector is selected from the local motion vectors of the first image using sensor data from the motion sensor.

[0024] The step of selecting the effective local motion vector may include the following steps: when the first image is an image captured at normal magnification, a sensor vector is generated based on the sensor data, and among the local motion vectors of the first image, the following local motion vector is selected as the effective local motion vector: the difference between the size and direction of the local motion vector and the size and direction of the sensor vector is less than or equal to a critical value.

[0025] The method may further include the step of detecting a motion region in the first image after the jitter has been corrected.

[0026] Invention Effects

[0027] The image processing apparatus of the present invention can provide a stable image with shake-free image even when there is a moving object corresponding to the image shake. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating the internal structure of an image processing apparatus according to one embodiment.

[0029] Figure 2 It is used for explanation Figure 1 The image shows a diagram of feature point extraction from the image processing device.

[0030] Figure 3 and Figure 4 It is used for explanation Figure 1 The image shows a diagram of the local motion vector extraction from the image processing device.

[0031] Figure 5 It is used for explanation Figure 1The image processing device shown is used to remove local motion vectors.

[0032] Figure 6 It is a general description Figure 1 A flowchart of the image processing method of the image processing apparatus shown.

[0033] Figure 7 It is a general description Figure 6 The flowchart for step 651.

[0034] Figure 8 It is a general description Figure 6 The flowchart for step 652.

[0035] Figure 9 and Figure 10 This is a diagram used to illustrate step 652. Detailed Implementation

[0036] The following content is merely illustrative of the principles of the invention. Therefore, although not explicitly described or shown in this specification, those skilled in the art can invent various means to implement the principles of the invention and which are included within the concept and scope of the invention. Furthermore, it should be understood that all terms and embodiments listed in this specification are intended only to clarify the concept of the invention, and such specifically listed embodiments and states are not limiting. Additionally, it should be understood that not only the principles, ideas, and embodiments of the invention, but also all detailed descriptions of specific embodiments are intended to include structural and functional equivalents of these matters. Furthermore, these equivalents should be understood to include not only currently known equivalents but also equivalents to be developed in the future, i.e., all elements invented in a manner that performs the same function independently of structure.

[0037] Therefore, the functionality of the various elements shown in the figures, including processors or functional blocks represented by similar concepts, can be provided using dedicated hardware and hardware capable of executing software in association with appropriate software. When provided by a processor, functionality can be provided by a single dedicated processor, a single shared processor, or multiple separate processors, some of which may be shared. Furthermore, the use of terms presented as processors, controls, or similar concepts should not be construed as specifically referring to hardware capable of executing software, but should be understood to implicitly include, but is not limited to, digital signal processor (DSP) hardware, read-only memory (ROM), random access memory (RAM), and non-volatile memory for storing software. Other commonly used hardware may also be included.

[0038] The aforementioned objectives, features, and advantages will become more apparent from the following description in relation to the accompanying drawings. In describing the invention, detailed descriptions or brief descriptions of relevant known techniques will be omitted or omitted if it is determined that such detailed descriptions would unnecessarily obscure the spirit of the invention.

[0039] The terminology used in the following embodiments is for illustrative purposes only and is not intended to limit the invention. Unless the context clearly indicates otherwise, singular expressions also include plural expressions. In the following embodiments, it should be understood that terms such as "comprising" or "possessing" are used to specify the presence of a feature, number, step, operation, constituent element, component, or combination of these elements as described in the specification, without excluding the possibility of the presence or addition of more than one other feature, number, step, operation, constituent element, component, or combination of these elements.

[0040] In the following embodiments, terms such as "first," "second," etc., may be used to describe various constituent elements, but the constituent elements should not be limited by the terms. The terms are used only to distinguish one constituent element from another.

[0041] The present invention according to preferred embodiments will now be described in detail with reference to the accompanying drawings.

[0042] Figure 1 This is a schematic diagram illustrating the internal structure of an image processing apparatus according to one embodiment. Figure 2 It is used for explanation Figure 1 The image shows a diagram of feature point extraction from the image processing device. Figure 3 and Figure 4 It is used for explanation Figure 1 The image shows a diagram of the local motion vector extraction from the image processing device. Figure 5 It is used for explanation Figure 1 The image processing device shown is used to remove local motion vectors. Figure 6 It is a general description Figure 1 A flowchart of the image processing method of the image processing apparatus shown.

[0043] Reference Figure 1 In one embodiment, the image processing apparatus 1 can be connected to the image sensor 60, the motion sensor 70, and the storage unit 80 via wired and / or wireless means. The image processing apparatus 1 can communicate with the image sensor 60, the motion sensor 70, and the storage unit 80 via wired and / or wireless means. Wireless communication may include short-range wireless communication such as Zigbee, Bluetooth, Radio Frequency Identification (RFID), Near Field Communication (NFC), and infrared communication, as well as mobile communication or wireless network communication such as 3G, 4G (LTE), 5G, WiFi, Wibro, and WiMAX.

[0044] The image sensor 60 can be an image capturing device that uses photoelectric conversion devices such as charge-coupled devices (CCD) or complementary metal-oxide-semiconductors (CMOS) to convert light signals into electrical signals and generate a series of images.

[0045] Motion sensor 70 can generate motion data corresponding to the physical jitter of image sensor 60. Motion sensor 70 may include a gyroscope sensor or an accelerometer sensor. Motion sensor 70 can generate motion data by detecting the amount of physical rotation of image sensor 60. When motion sensor 70 is a gyroscope sensor, the motion data may be the angular velocity of image sensor 60.

[0046] The storage unit 80 may store programs for processing and controlling the image processing apparatus 1, and may also store input or output data (e.g., images). The storage unit 80 may include internal memory and / or external storage media such as an SD card. The storage unit 80 may include web storage, a cloud server, or other storage devices that perform storage functions over a network.

[0047] When the image sensor 60 vibrates due to user hand tremors, wind, external impacts, etc., rotation and / or translation occur. Distortion may appear in the captured image due to the vibration of the image sensor 60. An embodiment of the image processing apparatus 1 of the present invention can correct image distortion caused by the vibration of the image sensor 60 and output a stable image. When a motion sensor 70 is provided within the image processing apparatus 1, the motion sensor 70 senses the vibration of the image processing apparatus 1, and the image processing apparatus 1 corrects the image distortion caused by the vibration of the image processing apparatus 1 and outputs a stable image.

[0048] An image processing apparatus 1 in one embodiment may include a feature extractor 10, a first motion vector extractor 20, a second motion vector extractor 30, an image stabilizer 40, and a motion area detector 50. Figure 1In this embodiment, the first motion vector extractor 20 and the second motion vector extractor 30 are shown as separate components. However, in other embodiments, the first motion vector extractor 20 and the second motion vector extractor 30 may be integrated into a single component, such as a motion vector extractor. Hereinafter, reference will be made to... Figure 6 Please provide an explanation.

[0049] The feature point extractor 10 can extract feature points from the current image IMc (first image) input from the image sensor 60 (S61). For example... Figure 2 As shown, the feature point extractor 10 can segment the current image IMc into multiple sub-blocks SB. The feature point extractor 10 can extract representative feature points FP from each sub-block SB. There are no particular limitations on the feature point extraction method; for example, the feature point extractor 10 can utilize the Harris corner detection method, the scale-invariant feature transform (SIFT) algorithm, the speedup robust feature (SURF) algorithm, etc., to extract feature points such as corners, edges, contours, and line intersections from the current image IMc. The feature point extractor 10 can detect the feature point with the highest eigenvalue among at least one feature point extracted from each sub-block SB as the representative feature point FP.

[0050] The first motion vector extractor 20 can extract the local motion vector LMV (S63) of the representative feature point FP for each sub-block SB. The first motion vector extractor 20 can extract the local motion vector LMV of the representative feature point FP by comparing the current image IMc with the previous image IMp (second image). For example... Figure 3 As shown, in the current image IMc, a first region A1 of size M×N is defined centered on the feature point FP. In the previous image Imp, a search region SA corresponding to the first region A1 is defined. The search region SA can be set to a size larger than the first region A1. For example, the search region SA can be set to a size of (M+P)×(N+P). Here, M, N, and P can represent the number of pixels, respectively. The previous image is an image input by the image sensor 60 before the current image, and can be an image input immediately before the current image.

[0051] The first motion vector extractor 20 can identify the region most similar to the first region A1 as the second region A2 within the search region SA of the previous image IMP. The first motion vector extractor 20 can calculate the vector from the corresponding feature point FP' of the second region A2 to the representative feature point FP of the first region A1, which is used as the local motion vector LMV of the representative feature point FP. Figure 4 The local motion vector LMV for the representative feature point FP of each sub-block SB is shown.

[0052] The embodiments of the present invention utilize a block matching method to extract motion vectors from feature points, thereby reducing the amount of computation.

[0053] The first motion vector extractor 20 can verify the local motion vector (LMV) extracted from the current image IMc (S65). The first motion vector extractor 20 selects valid local motion vectors from the local motion vectors (LMV) extracted from the current image IMc to verify the local motion vector (LMV). The first motion vector extractor 20 can use different local motion vector (LMV) verification algorithms based on the zoom ratio of the image sensor 60. For this purpose, the image processing apparatus 1 can determine the zoom ratio of the image sensor 60.

[0054] When the image sensor 60 is shooting at normal magnification, the first motion vector extractor 20 can use the motion data output by the motion sensor 70 to verify the local motion vector LMV (S651). That is, when the image sensor 60 is operating at normal magnification, the first motion vector extractor 20 simultaneously uses the image output by the image sensor 60 and the motion data output by the motion sensor 70 to verify the local motion vector LMV.

[0055] When the image sensor 60 captures images at high magnification, the first motion vector extractor 20 uses motion region data output by the motion region detector 50 to verify the local motion vector LMV (S652). That is, the first motion vector extractor 20 can use the image output by the image sensor 60 and the motion region data output by the motion region detector 50 to verify the local motion vector LMV. The motion region data can be motion region data from previous images. Previous images can be images corrected in the image stabilizer 40. The distinction between normal magnification and high magnification can vary depending on the user's settings. For example, zoom magnification above 40x can be set as high magnification, while low and medium magnification zoom magnifications can be set as normal magnification. Here, normal magnification and high magnification can refer to optical zoom magnification.

[0056] like Figure 5 As shown, the first motion vector extractor 20 can detect valid local motion vectors (LMVs) in the current image IMc by verifying the local motion vectors (LMVs) and removing LMVs corresponding to the motion of the subject rather than those corresponding to the motion of the image sensor 60. The verification of local motion vectors (LMVs) will be described below.

[0057] The second motion vector extractor 30 can extract the global motion vector GMV based on the effective local motion vectors (LMV) detected from the current image IMc (S67). The second motion vector extractor 30 can extract the global motion vector GMV based on the effective local motion vectors (LMV) detected from the current image IMc acquired at normal magnification (S67a). The second motion vector extractor 30 can extract the global motion vector GMV based on the effective local motion vectors (LMV) detected from the current image IMc acquired at high magnification (S67b).

[0058] The second motion vector extractor 30 can calculate a histogram of effective local motion vectors (LMV) and extract the global motion vector (GMV) based on the calculated histogram. Embodiments of the present invention are not limited to this; the global motion vector (GMV) can be extracted using various methods. For example, the average or median of effective local motion vectors (LMV) can be extracted as the global motion vector (GMV).

[0059] Image stabilizer 40 can reference the global motion vector GMV to correct the jitter of the current image IMc, thereby outputting the corrected current image IMcc (S69). Image stabilizer 40 can reference the global motion vector GMV extracted from the current image IMc acquired at normal magnification to output the corrected current image IMcc (S69a). Image stabilizer 40 can reference the global motion vector GMV extracted from the current image IMc acquired at high magnification to output the corrected current image IMcc (S69b).

[0060] The motion region detector 50 can receive the corrected current image IMcc and detect motion regions from it. The motion region detector 50 can utilize a Gaussian Mixture Model (GMM) to detect motion regions from the corrected current image IMcc. GMM has a strong advantage in handling subtle background jitter (dynamic background). However, embodiments of the invention are not limited to this; motion regions can be detected through various methods, such as detecting motion regions from the difference image between the reference image and the corrected current image IMcc. The motion region detector 50 can output the coordinates of the motion regions detected in the corrected current image IMcc to the first motion vector extractor 20 for detecting valid local motion vectors (LMVs) in the next image. The coordinates of the motion regions may include the coordinates of the foreground.

[0061] The first motion vector extractor 20 can use the coordinates of the motion region of the current image IMcc, which has been corrected and received from the motion region detector 50, to verify the local motion vector LMV of the next image. That is, the first motion vector extractor 20 can use the coordinates of the motion region of previous images to verify the local motion vector LMV of the current image.

[0062] First, we will explain the verification of the Local Motion Vector (LMV) when the image sensor 60 is shooting at normal magnification. Figure 7 It is a general description Figure 6 The flowchart for step 651. See below for reference. Figure 1 and Figure 7 Please provide an explanation.

[0063] The first motion vector extractor 20 receives sensor data from the motion sensor 70 and transforms the sensor data to generate a sensor vector (S71). The sensor data can be the physical rotation of the image sensor 60 detected by the motion sensor 70, such as the angular velocity measured in a gyroscope sensor. The first motion vector extractor 20 can transform the sensor data into pixel displacement values ​​to generate the sensor vector.

[0064] The first motion vector extractor 20 compares the local motion vector (LMV) extracted from the current image IMc with the sensor vector, thereby selecting a valid local motion vector (LMV) (S73). When the difference between the direction and size of the sensor vector and the direction and size of the local motion vector (LMV) is below a critical value, the first motion vector extractor 20 determines that the local motion vector (LMV) is valid. When the difference between the direction and size of the sensor vector and the direction and size of the local motion vector (LMV) deviates from the critical value (i.e., exceeds the critical value), the first motion vector extractor 20 determines that the local motion vector (LMV) represents motion caused by the movement of the subject being photographed, and removes the local motion vector (LMV).

[0065] Next, the verification of the Local Motion Vector (LMV) when the image sensor 60 captures images at high magnification will be explained. Figure 8 It is a general description Figure 6 The flowchart for step 652. Figure 9 and Figure 10 This is a diagram used to illustrate step 652. See below for reference. Figure 1 and Figure 8 Please provide an explanation.

[0066] When there is a moving object in the image, the first motion vector extractor 20 receives the coordinates of at least one motion region from the motion region detector 50 in the previous image. In the current image IMc, using the received coordinates of the motion region, it searches for the foreground region BLOB corresponding to the motion region and sets a first target region TA1 smaller than the foreground region BLOB within the foreground region BLOB (S81). Figure 9 This example illustrates setting a first target region TA1 within two foreground regions BLOB1 and BLOB2. The horizontal and vertical dimensions of the first target region TA1 can be 2D smaller than the horizontal and vertical dimensions of the foreground regions BLOB1 and BLOB2, respectively. Here, D can be the maximum value of the local motion vector LMV of the current image IMc. The size of the first target region TA1 can vary depending on the size of the foreground regions BLOB1.

[0067] The first motion vector extractor 20 can calculate the average value of the local motion vector LMV existing in the first target region TA1 (S83). The first motion vector extractor 20 can calculate the average value of the x-direction component and the average value of the y-direction component of the local motion vector LMV.

[0068] Equation (1) is a formula for calculating the average value (μ) of the x-direction component of the local motion vector LMV existing in the first target region TA1 of the i-th foreground region BLOB. Equation (2) is a formula for calculating the average value (μ) of the y-direction component of the local motion vector LMV existing in the first target region TA1 of the i-th foreground region BLOB. M is the number of local motion vectors LMV existing in the first target region TA1.

[0069]

[0070]

[0071] The first motion vector extractor 20 can set a second target region TA2 (S85) in the current image IMc that is larger than the foreground region BLOB. Figure 10 This example illustrates setting a second target region TA2 within two foreground regions BLOB1 and BLOB2. The horizontal and vertical dimensions of the second target region TA2 can be 2D larger than the horizontal and vertical dimensions of the foreground regions BLOB1 and BLOB2, respectively. The size of the second target region TA2 can vary depending on the size of the foreground regions BLOB1 and BLOB2.

[0072] The first motion vector extractor 20 compares the size of the local motion vectors (LMVs) existing in the second target region TA2 with the average value of the local motion vectors (LMVs) calculated in step 83, thereby selecting valid local motion vectors (LMVs) (S87). When the difference between the average value and the size of the local motion vector (LMV) is above a critical value, the first motion vector extractor 20 determines that the local motion vector (LMV) is valid. When the difference between the average value and the size of the local motion vector (LMV) is less than the critical value, the first motion vector extractor 20 determines that the local motion vector (LMV) represents motion caused by the movement of the subject being photographed, and removes the local motion vector (LMV).

[0073] Referring to equation (3), when the difference between the size I of the x-direction component and the y-direction component of the local motion vector LMV existing in the second target area TA2 of the i-th foreground area BLOB and the average value (μ) calculated in step 83 is less than the critical value L, the first motion vector extractor 20 judges the local motion vector LMV as the motion vector M of the photographed object, and classifies the others as valid local motion vectors LMV.

[0074]

[0075] The image processing apparatus 1 can be configured to capture, create, process, convert, scale, encode, decode, transmit, store, and display digital images and / or video sequences. The image processing apparatus 1 can be implemented as a device such as a wireless communication device, a personal digital assistant (PDA), a laptop computer, a desktop computer, a portable camcorder, a digital camera, CCTV, an action camera, a digital recording device, a network-connected digital television, a mobile phone, a cellular phone, a satellite phone, a camera phone, or a two-way communication device.

[0076] In the above embodiments, the image sensor 60, motion sensor 70, and storage unit 80 are shown as separate components from the image processing apparatus 1. However, in other embodiments, at least one of the image sensor 60, motion sensor 70, and storage unit 80 may be included in the image processing apparatus 1 as part of the image processing apparatus 1.

[0077] With the development of optical technology, CCTV surveillance cameras and other devices support high-magnification zoom. At high magnification, even slight jitter that might render the motion sensor output ineffective can result in significant motion in the actual image. In cases of slight jitter, image stabilization methods using motion sensors such as gyroscopes can be difficult, and image-based stabilization may lead to erroneous actions such as misinterpreting the motion of large moving objects or multiple objects in the same direction as global motion within the image.

[0078] In embodiments of the present invention, at normal magnification, more accurate and effective Local Motion Vector (LMV) can be extracted using sensor data from a motion sensor. At high magnification, when sensor data from the motion sensor is very limited, effective LMV can be extracted based solely on the image. In other embodiments, even at high magnification, if sufficient sensor data from the motion sensor is ensured, effective LMV can be extracted using the motion sensor data, just as at normal magnification.

[0079] In embodiments of the present invention, at high magnification, in response to a moving object (a moving subject), a Gaussian mixture model (GMM) or similar method is used to detect motion regions for an image that has been stabilized through correction. Furthermore, in the local motion vectors (LMVs) of the current image, by excluding LMVs that exhibit similar patterns to those in motion regions of previous images and their adjacent regions, motion vectors corresponding to moving objects are excluded, thereby enabling the extraction of more stable global motion vectors.

[0080] The image processing apparatus and image processing method of the present invention can provide stable image signals to various intelligent image monitoring systems such as aviation and military facilities, ports, roads, bridges, subways, buses, building roofs, stadiums, parking lots, automobiles and mobile devices, and robots.

[0081] The present invention has been described with reference to one embodiment shown in the accompanying drawings, but this is merely an example, and those skilled in the art will understand that various modifications and equivalent embodiments can be made therefrom. Therefore, the true scope of protection of the present invention should be determined by the appended claims.

Claims

1. An image processing apparatus, in, include: A feature point extractor detects multiple feature points from a first image input by an image sensor; A first motion vector extractor extracts local motion vectors for the multiple feature points, and uses different algorithms to verify the extracted local motion vectors according to the zoom ratio of the image sensor, thereby selecting effective local motion vectors from the extracted local motion vectors. A second motion vector extractor extracts a global motion vector using the selected effective local motion vectors; and An image stabilizer corrects for jitter in the first image based on the global motion vector. Wherein, when the first image is an image captured at a high magnification of a set magnification or higher, the first motion vector extractor selects effective local motion vectors from the local motion vectors of the first image by utilizing motion regions detected in a second image input from the image sensor prior to the first image. When the first image is a normal magnification image captured at a magnification lower than the set magnification, the first motion vector extractor uses sensor data from the motion sensor to select the effective local motion vector from the local motion vectors of the first image.

2. The image processing apparatus according to claim 1, wherein, When the first image is an image captured at a high magnification of more than a set magnification, the first motion vector extractor, Within the foreground region of the first image corresponding to the motion region detected in the second image, a first target region smaller than the foreground region is defined, and the average size of the local motion vectors existing within the first target region is calculated. In the first image, a second target region larger than the foreground region is defined, and among the local motion vectors existing in the second target region, the following local motion vectors are selected as the effective local motion vectors: the difference between the size of the local motion vector and the average size is greater than or equal to a critical value.

3. The image processing apparatus according to claim 2, wherein, The horizontal and vertical dimensions of the first target region are smaller than those of the foreground region, respectively, and the difference between them is twice the maximum value of the local motion vector dimensions in the first image.

4. The image processing apparatus according to claim 2, wherein, The horizontal and vertical dimensions of the second target region are larger than those of the foreground region, respectively, and the difference between them is twice the maximum value of the local motion vector dimensions in the first image.

5. The image processing apparatus according to claim 2, wherein, The average size of the local motion vector includes the average size of the x-direction component and the average size of the y-direction component.

6. The image processing apparatus according to claim 1, wherein, When the first image is an image captured at a normal magnification lower than a set magnification, the first motion vector extractor generates a sensor vector based on the sensor data, and selects a local motion vector from the local motion vectors of the first image as the effective local motion vector: the difference between the size and direction of the local motion vector and the size and direction of the sensor vector is less than or equal to a critical value.

7. The image processing apparatus according to claim 1, wherein, It also includes a motion region detector to detect motion regions in the first image after the jitter has been corrected.

8. An image processing apparatus, wherein, include: A feature point extractor detects multiple feature points from an image input from an image sensor. The motion vector extractor extracts the local motion vectors of the multiple feature points. A motion region detector detects motion regions from the image, and An image stabilizer that corrects image jitter using the local motion vector and the motion region; The motion vector extractor, Within the foreground region corresponding to the motion region detected in previous images in the current image, a first target region smaller than the foreground region is defined, and the average size of the local motion vectors existing within the first target region is calculated. In the current image, a second target region larger than the foreground region is defined. Among the local motion vectors existing in the second target region, local motion vectors whose size and the average size difference are less than a critical value are removed, and local motion vectors whose difference is greater than or equal to the critical value are selected.

Images