Traffic information collection system, real-time image stabilization device, and real-time image stabilization method

Abstract

The present invention relates to a traffic information collection system, a real-time image stabilization device, and a real-time image stabilization method, wherein an image including a road is acquired through an image acquisition device, multiple feature points extracted from a first frame and a second frame are matched so that movement information is obtained, image shake information is obtained on the basis of the movement information, and the image is corrected on the basis of the image shake information.

Description

Traffic information collection system, real-time image stabilization device, and real-time image stabilization method

[0001] Embodiments of the present disclosure relate to a traffic information collection system, a real-time image stabilization device, and a real-time image stabilization method.

[0002] Traffic information collection systems can collect traffic data based on video. If the video captured by the system's video capture device is shaky, errors may occur during object tracking, traffic volume analysis, and other processes, or the collection of accurate traffic data may be compromised.

[0003] Traditionally, camera shake issues in other industries have been addressed primarily by mounting expensive sensors and moving the lens during the filming process, or by post-processing the entire video after it's been captured. These methods are unsuitable for the transportation industry, where numerous cameras are fixed in various locations and real-time processing is crucial.

[0004] Embodiments of the present disclosure can provide a traffic information collection system, a real-time image stabilization device, and a real-time image stabilization method that provide a real-time image stabilization function that enables accurate traffic information collection.

[0005] Embodiments of the present disclosure can provide a traffic information collection system, a real-time image stabilization device, and a real-time image stabilization method that can recognize image shaking and perform image correction accordingly.

[0006] A traffic information collection system according to embodiments of the present disclosure may include an image acquisition device that acquires an image including a road, a real-time image stabilization device that acquires image shake information by comparing a first frame and a second frame in the image, and corrects the image based on the image shake information.

[0007] The image acquisition device may be fixed.

[0008] A real-time image stabilization device may include a feature point extraction unit that extracts a plurality of feature points from a first frame and a plurality of feature points from a second frame, a feature point matching unit that matches a plurality of feature points extracted from the first frame with a plurality of feature points extracted from the second frame, a feature point movement information acquisition unit that calculates movement information for each of a plurality of feature point sets including two feature points matched by the feature point matching unit, an image shake information identification unit that identifies image shake information based on the movement information, and an image shake correction unit that corrects an image based on the image shake information.

[0009] A real-time image stabilization device according to embodiments of the present disclosure may include an input unit for receiving an image, a real-time image stabilization processing unit for obtaining image shake information by comparing a first frame and a second frame in the image, and for correcting the image based on the image shake information.

[0010] The real-time image stabilization processing unit may include a feature point extraction unit that extracts a plurality of feature points from a first frame and a plurality of feature points from a second frame, a feature point matching unit that matches a plurality of feature points extracted from the first frame with a plurality of feature points extracted from the second frame, a feature point movement information acquisition unit that calculates movement information for each of a plurality of feature point sets including two feature points matched by the feature point matching unit, an image shake information identification unit that identifies image shake information based on the movement information, and an image shake correction unit that corrects an image based on the image shake information.

[0011] A real-time image stabilization method according to embodiments of the present disclosure may include a step of acquiring an image including a road, a step of acquiring image shake information by comparing a first frame and a second frame in the image, and a step of correcting the image based on the image shake information.

[0012] The step of obtaining image shake information may include a step of extracting a plurality of feature points from a first frame, a step of extracting a plurality of feature points from a second frame, a step of matching a plurality of feature points extracted from the first frame with a plurality of feature points extracted from the second frame, a step of calculating movement information for each of a plurality of feature point sets including two feature points matched by a feature point matching unit, and a step of obtaining image shake information based on the movement information.

[0013] According to embodiments of the present disclosure, a traffic information collection system, a real-time image stabilization device, and a real-time image stabilization method that provide a real-time image stabilization function enabling accurate traffic information collection can be provided.

[0014] According to embodiments of the present disclosure, embodiments of the present disclosure can provide a traffic information collection system, a real-time image stabilization device, and a real-time image stabilization method capable of recognizing image shake and performing image correction accordingly.

[0015] FIG. 1 illustrates a traffic information collection system according to embodiments of the present disclosure.

[0016] Figure 2 shows the installation environment of an image acquisition device in a traffic information collection system according to embodiments of the present disclosure.

[0017] FIG. 3 is a block diagram of a real-time image stabilization device according to embodiments of the present disclosure.

[0018] FIG. 4 is a diagram for explaining feature point extraction and matching between frames during real-time image stabilization according to embodiments of the present disclosure.

[0019] FIG. 5 shows an image before correction by a real-time image stabilization device according to embodiments of the present disclosure.

[0020] FIG. 6 shows an image after correction by a real-time image stabilization device according to embodiments of the present disclosure.

[0021] FIG. 7 is a flowchart of a real-time image stabilization method according to embodiments of the present disclosure.

[0022] FIG. 8 is a detailed flowchart of an image shake information acquisition step of a real-time image stabilization method according to embodiments of the present disclosure.

[0023] Embodiments of the present disclosure may disclose a real-time image stabilization method for solving a “real-time video shaking problem” through software, a real-time image stabilization device performing the same, and a traffic information collection system including the real-time image stabilization device.

[0024] A real-time image stabilization device according to embodiments of the present disclosure first extracts feature points from a reference frame within an image acquired by an image acquisition device that may be a camera, and then, in real time, whenever a current frame (hereinafter referred to as a “second frame”) is received, compares the feature points with the reference frame (a frame preceding the current frame, also referred to as a “first frame”) and matches them.

[0025] That is, the real-time image stabilization device according to the embodiments of the present disclosure can compare and match feature points of a current frame with feature points of a reference frame.

[0026] At this time, considering that the image acquisition device, which may be a camera, is fixed, the matching performance can be greatly improved by adding a condition so that the two feature points are matched when the distance between the feature points (the feature point of the first frame and the feature point of the second frame) is close enough to be within a predetermined threshold value.

[0027] Thereafter, the real-time image stabilization device according to the embodiments of the present disclosure can perform shake correction (video stabilization) on the image based on movement information (e.g., movement amount, movement direction) of matched feature points.

[0028] A real-time image stabilization device according to embodiments of the present disclosure can enable rapid, consistent, and high-performance image stabilization by correcting image shake based on feature point extraction and matching.

[0029] The video stabilization achieved in this way works effectively against both large and small shaking. This allows for accurate recognition of objects such as vehicles and pedestrians without missing them, and contributes to building a more reliable system by reliably and accurately tracking these objects, thereby minimizing errors in traffic information. Furthermore, real-time processing using only software allows for easy integration with various algorithms and offers excellent expandability to other industries.

[0030] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When adding reference numerals to components in each drawing, the same components may have the same numerals as much as possible even if they are shown in different drawings. In addition, when describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description may be omitted. When "includes," "has," "consists of," etc. are used in this specification, other parts may be added unless "only" is used. When a component is expressed in the singular, it may include a plural unless there is a special explicit description.

[0031] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the present disclosure. These terms are only intended to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by the terms.

[0032] In a description of the positional relationship of components, when it is described that two or more components are "connected," "combined," or "connected," it should be understood that the two or more components may be directly "connected," "combined," or "connected," but that the two or more components may also be further "interposed" with another component to be "connected," "combined," or "connected." Here, the other component may be included in one or more of the two or more components that are "connected," "combined," or "connected" to each other.

[0033] In the description of the temporal flow relationship related to components, operation methods, or manufacturing methods, for example, when the temporal or flow relationship is described as “after”, “following”, “next to”, “before”, etc., it may also include cases where it is not continuous, unless “immediately” or “directly” is used.

[0034] Meanwhile, when numerical values ​​or corresponding information (e.g., levels, etc.) for components are mentioned, even without separate explicit description, the numerical values ​​or corresponding information may be interpreted as including an error range that may occur due to various factors (e.g., process factors, internal or external impact, noise, etc.).

[0035] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0036] Fig. 1 illustrates a traffic information collection system (100) according to embodiments of the present disclosure. Fig. 2 illustrates the installation environment of an image acquisition device (110) within a traffic information collection system (100) according to embodiments of the present disclosure.

[0037] Referring to FIG. 1, a traffic information collection system (100) according to embodiments of the present disclosure may include an image acquisition device (110) that acquires an image including a road, and a real-time image stabilization device (120) that acquires image shake information by comparing a first frame (FRAME1) and a second frame (FRAME2) within the image, and corrects the image based on the image shake information.

[0038] Here, the first frame (FRAME1) and the second frame (FRAME2) may be either immediately adjacent frames or temporally separated frames. That is, there may be no other frames between the first frame (FRAME1) and the second frame (FRAME2), or there may be at least one additional frame.

[0039] In embodiments of the present disclosure, the first frame (FRAME1) means a reference frame and may be the i-th frame, and the second frame (FRAME2) means a frame to be compared with the first frame (FRAME1) and may be the j-th frame (j=i+k, k is a natural number greater than or equal to 1).

[0040] Referring to FIG. 1, a traffic information collection system (100) according to embodiments of the present disclosure may further include a traffic information generation device (130) that generates traffic information on a road based on a corrected image and stores and outputs the generated traffic information. Here, the stored traffic information corresponds to traffic collection.

[0041] The image acquisition device (110) is fixed. For example, the image acquisition device (110) may be fixed on the road or on the outskirts of the road, or may be located at an elevated location in the middle of the road, for effective collection of road information.

[0042] As illustrated in FIG. 2, the image acquisition device (110) may be installed fixedly on a road structure (200) installed on the road. In this case, the image acquisition device (110) may be positioned at a high location in the middle of the road.

[0043] Due to the fixed position of the image acquisition device (110), the image acquisition device (110) may shake due to strong winds, etc. Due to the shaking of the image acquisition device (110), shaking of the image acquired by the image acquisition device (110) may also easily occur.

[0044] If shaking occurs in the image acquired by the image acquisition device (110), stability may be reduced and errors may occur in the process of analyzing object tracking, traffic volume, etc. using traffic information collected by the traffic information generation device (130).

[0045] Accordingly, the real-time image stabilization device (120) included in the traffic information collection system (100) according to embodiments of the present disclosure can obtain image shake information by comparing the first frame (FRAME1), which is a reference frame within the image, with the current frame (FRAME2), and correct the image based on the image shake information.

[0046] An image corrected by a real-time image stabilization device (120) may be an image from which image shaking has been removed.

[0047] The traffic information generation device (130) generates traffic information on the road based on a corrected image (corrected image), thereby generating accurate traffic information from a corrected image from which the influence of image shaking has been removed.

[0048] The image acquisition device (110), the real-time image stabilization device (120), and the traffic information generation device (130) may all be configured with different hardware. Alternatively, two or more of the image acquisition device (110), the real-time image stabilization device (120), and the traffic information generation device (130) may be integrated into a single piece of hardware.

[0049] That is, the image acquisition device (110), the real-time image stabilization device (120), and the traffic information generation device (130) may all be configured separately, or two of the image acquisition device (110), the real-time image stabilization device (120), and the traffic information generation device (130) may be configured as one, or the image acquisition device (110), the real-time image stabilization device (120), and the traffic information generation device (130) may all be configured as one.

[0050] Accordingly, the accuracy and stability of analysis of object tracking, traffic volume, etc. can be improved by utilizing traffic information.

[0051] Below, a real-time image stabilization device (120) according to embodiments of the present disclosure is described in detail.

[0052] FIG. 3 is a block diagram of a real-time image stabilization device (120) according to embodiments of the present disclosure, FIG. 4 is a drawing for explaining feature point extraction and matching between frames during real-time image stabilization according to embodiments of the present disclosure, FIG. 5 shows an image before correction by a real-time image stabilization device (120) according to embodiments of the present disclosure, and FIG. 6 shows an image after correction by a real-time image stabilization device (120) according to embodiments of the present disclosure.

[0053] Referring to FIG. 3, a real-time image stabilization device (120) according to embodiments of the present disclosure may include an input unit (310) for receiving an image, a real-time image stabilization processing unit (320) for correcting the image so that the influence of image shaking is removed, and an image storage unit (330) for storing the input image and / or the corrected image.

[0054] The real-time image stabilization processing unit (320) can obtain image shake information by comparing the first frame (FRAME1) and the second frame (FRAME2) within the image, and correct the image based on the image shake information.

[0055] The real-time image stabilization processing unit (320) may use a color image composed of 3D RGB before comparing the first frame (FRAME1) and the second frame (FRAME2) in the image, or may convert the color image composed of 3D RGB into a 1D black and white image and use it.

[0056] The real-time image stabilization processing unit (320) may include a feature point extraction unit (321), a feature point matching unit (322), a feature point movement information acquisition unit (323), an image shake information identification unit (324), and an image shake correction unit (325).

[0057] The feature point extraction unit (321) can extract a plurality of first feature points (FP1) from a first frame (FRAME1) and a plurality of second feature points (FP2) from a second frame (FRAME2). Here, the first frame (FRAME1) and the second frame (FRAME2) can be color images composed of three-dimensional RGB, or one-dimensional black-and-white images converted from color images composed of three-dimensional RGB.

[0058] The feature point matching unit (322) can match a plurality of first feature points (FP1) extracted from a first frame (FRAME1) by the feature point extraction unit (321) and a plurality of second feature points (FP2) extracted from a second frame (FRAME2).

[0059] The feature point movement information acquisition unit (323) can produce movement information for each of a plurality of feature point sets including two feature points (FP1, FP2) matched by the feature point matching unit (322).

[0060] The image shake information acquisition unit (324) can acquire image shake information based on the movement information produced by the feature point movement information acquisition unit (323).

[0061] The image shake correction unit (325) can correct an image based on the image shake information identified by the image shake information identification unit (324). The corrected image can be stored by the image storage unit (330).

[0062] Below, the operation of the internal components of the real-time image stabilization processing unit (320) is described in more detail.

[0063] The first frame (FRAME1) and the second frame (FRAME2) of FIG. 4 are images taken at the same location at different times, and many lines displayed between the first frame (FRAME1) and the second frame (FRAME2) are matching lines of feature points.

[0064] Referring to FIG. 4, the left end of each line corresponds to the first feature point (FP1) extracted from the first frame (FRAME1), and the right end of each line corresponds to the second feature point (FP2) extracted from the second frame (FRAME2). The fact that the first feature point (FP1) and the second feature point (FP2) are connected by a line means that the first feature point (FP1) and the second feature point (FP2) are matched.

[0065] Referring to FIGS. 3 and 4, the feature point extraction unit (321) can extract color change points within the first frame (FRAME1) as a plurality of first feature points (FP1).

[0066] For example, the color change points may include at least one of a hue change point, a brightness change point, and a saturation change point.

[0067] When a second frame (FRAME2) is input after a first frame (FRAME1), the feature point extraction unit (321) can extract color change points within the second frame (FRAME2) as a plurality of second feature points (FP2).

[0068] For example, color change points may correspond to boundary points between fixed objects around the road and the background.

[0069] The feature point extraction unit (321) can extract boundary points between fixed objects around the road and the background in the first frame (FRAME1) as a plurality of first feature points (FP1), and can extract boundary points between fixed objects around the road and the background in the second frame (FRAME2) as a plurality of second feature points (FP2).

[0070] As another example, color change points may correspond to boundaries between fixed objects around a road.

[0071] The feature point extraction unit (321) can extract boundary points between fixed objects around the road as a plurality of first feature points (FP1) in the first frame (FRAME1), and can extract boundary points between fixed objects around the road as a plurality of second feature points (FP2) in the second frame (FRAME2).

[0072] A fixed object around a road can be anything that is fixed and does not move around the road, such as a road sign, a traffic light, a roadside structure, a building, or a tree.

[0073] The background around the road may include the sky, mountains, or large buildings.

[0074] Referring to FIGS. 3 and 4, in order to accurately obtain image shake information, feature point matching between two frames (FRAME1, FRAME2) must be performed accurately, and for this purpose, the matched features (FP1, FP2) must be feature points related to a fixed object.

[0075] Accordingly, when matching a plurality of first feature points (FP1) extracted from a first frame (FRAME1) and a plurality of second feature points (FP2) extracted from a second frame (FRAME2), the feature point matching unit (322) matches the first feature points (FP1) and the second feature points (FP2) related to fixed objects (e.g., road structures, road signs, buildings, trees, etc.) and does not match the first feature points (FP1) and the second feature points (FP2) related to moving objects (e.g., vehicles, pedestrians, etc.).

[0076] Referring to FIGS. 3 and 4, the feature point matching unit (322) can match a plurality of first feature points (FP1) extracted from a first frame (FRAME1) and a plurality of second feature points (FP2) extracted from a second frame (FRAME2) based on the distance between the feature points.

[0077] For example, when a first feature point (FP1) is extracted from a first frame (FRAME1) and a second feature point (FP2) is extracted from a second frame (FRAME2), the feature point matching unit (322) may match the first feature point (FP1) and the second feature point (FP2) as feature points for the same point if the distance between the first feature point (FP1) and the second feature point (FP2) is less than or equal to a threshold value, and may not match the first feature point (FP1) and the second feature point (FP2) as feature points for the same point if the distance between the first feature point (FP1) and the second feature point (FP2) exceeds the threshold value.

[0078] Referring to FIGS. 3 and 4, the feature point matching unit (322) can exclude two feature points (i.e., the first feature point (FP1) and the second feature point (FP2) corresponding to a vehicle or a pedestrian) whose distance is greater than a threshold value from the matching target.

[0079] Referring to FIGS. 3 and 4, the feature point matching unit (322) can exclude feature points (e.g., vehicles, pedestrians, etc.) that exist only in one of the first frame (FRAME1) and the second frame (FRAME2) from the matching target.

[0080] Referring to FIG. 5, when image shaking occurs in the image before correction, the first feature point (FP1) in the first frame (FRAME1) is located at a point that is a first distance (L1) from the top, but the second feature point (FP2) matching the first feature point (FP1) in the second frame (FRAME2) is not located at a point that is a first distance (L1) from the top.

[0081] Within the second frame (FRAME2), the second feature point (FP2) matching the first feature point (FP1) is located at a point that is a certain distance (ΔL) further downward (D1) than a point that is a first distance (L1) from the top.

[0082] That is, the second feature point (FP2) matching the first feature point (FP1) within the second frame (FRAME2) is located at a distance (L1+ΔL) from the top, which is the sum of the first distance (L1) and the movement distance (ΔL).

[0083] Referring to FIG. 5, the feature point movement information acquisition unit (323) can generate movement information for each of a plurality of feature point sets including two feature points (FP1, FP2) matched by the feature point matching unit (322). Here, the movement information for the feature point set including the matched first feature point (FP1) and second feature point (FP2) can include a movement amount (ΔL) and a movement direction (D1). Here, the movement direction (D1) can include at least one of a vertical component (vertical component) and a horizontal component (horizontal component).

[0084] Referring to FIG. 5, the image shake information identification unit (324) can identify the image shake size included in the image shake information based on the movement amount (ΔL) included in the movement information for each of a plurality of feature point sets.

[0085] The size of the image shake (also called the degree of image shake) can be proportional to the amount of movement (ΔL).

[0086] For example, the image shake magnitude may be the average value of the displacement (ΔL) for each of the plurality of feature point sets. As another example, the image shake magnitude may be the most frequent value among the displacements (ΔL) for each of the plurality of feature point sets. As another example, the image shake magnitude may be the maximum value among the displacements (ΔL) for each of the plurality of feature point sets. As another example, the image shake magnitude may be a value calculated according to a predetermined formula based on the displacements (ΔL) for each of the plurality of feature point sets.

[0087] Referring to FIG. 5, the image shake information identification unit (324) can identify the image shake direction included in the image shake information based on the movement direction (D1) included in the movement information. Here, the image shake direction can correspond to the movement direction (D1).

[0088] For example, the direction of image shake may be the most frequent movement direction (D1) among the movement directions (D1) for each of the plurality of feature point sets. As another example, the direction of image shake may be the movement direction (D1) corresponding to the most frequent value among the movement amounts (ΔL) for each of the plurality of feature point sets. As yet another example, the magnitude of image shake may be the movement direction (D1) corresponding to the maximum value among the movement amounts (ΔL) for each of the plurality of feature point sets.

[0089] The movement information for each of the plurality of feature point sets may correspond to the movement information of the image acquisition device (110). Each of the plurality of feature point sets includes a first feature point (FP1) of a first frame (FRAME1) and a second feature point (FP2) of a second frame (FRAME2) that match each other. For example, the movement information for each of the plurality of feature point sets may include at least one of a movement amount (ΔL) and a movement direction (D1) for each of the plurality of feature point sets. For example, the movement information of the image acquisition device (110) may include at least one of a movement amount and a movement direction of the image acquisition device (110).

[0090] By the feature point extraction unit (321), a plurality of first feature points (FP1) extracted from the first frame (FRAME1) and a plurality of second feature points (FP2) extracted from the second frame (FRAME2) can correspond to color change points.

[0091] By the feature point extraction unit (321), a plurality of first feature points (FP1) extracted from the first frame (FRAME1) and a plurality of second feature points (FP2) extracted from the second frame (FRAME2) may correspond to boundary points between a fixed object around the road and the background, or may correspond to boundary points between a first fixed object around the road and a second fixed object.

[0092] In the following, for convenience of explanation, the image shake size is assumed to be ΔL, and the image shake direction is assumed to be D1.

[0093] Referring to FIG. 6, the image shake correction unit (325) sets the direction opposite to the image shake direction (D1) included in the image shake information as the correction direction (D2), sets the correction size (ΔL) corresponding to the image shake size (ΔL) included in the image shake information, and can correct the image based on the set correction direction (D2) and correction size (ΔL).

[0094] The image shake correction unit (325) may include, when processing an image transformation for image correction, adding or subtracting a constant value corresponding to the amount of movement (amount of change) in the vertical component and / or horizontal component, and in practice, may perform geometric transformation processing such as distortion or shape deformation by multiplying the image data by a predefined nXm matrix (e.g., 2X3 matrix).

[0095] Accordingly, as shown in FIG. 6, it can be confirmed that the position of the second feature point (FP2) in the second frame (FRAME2) is normally recovered.

[0096] In the second frame (FRAME2) of the image before correction, the second feature point (FP2) was located at a point (L1+ ΔL) away from the top of the screen, and was incorrectly positioned due to image shaking.

[0097] However, through image correction, the second feature point (FP2') in the second frame (FRAME2) of the corrected image is located at a point (L1) away from the top of the screen. In other words, the image shake is corrected through image correction, and the position of the second feature point (FP2) in the second frame (FRAME2) can be normalized.

[0098] A traffic information generation device (130) can generate various traffic information by recognizing and tracking objects (e.g., vehicles, pedestrians, etc.) through images.

[0099] Generating traffic information using images obtained through the aforementioned real-time image correction reduces object recognition errors, enabling more accurate object recognition and more precise and stable tracking of recognized objects. This allows for the generation of improved, more meaningful traffic information without errors.

[0100] The real-time image stabilization device (120) according to the embodiments of the present disclosure described above can be implemented as a server.

[0101] The real-time image stabilization method according to the embodiments of the present disclosure described above is briefly described again below.

[0102] FIG. 7 is a flowchart of a real-time image stabilization method according to embodiments of the present disclosure, and FIG. 8 is a detailed flowchart of an image shake information acquisition step of a real-time image stabilization method according to embodiments of the present disclosure.

[0103] Referring to FIG. 7, a real-time image stabilization method according to embodiments of the present disclosure may include a step of acquiring an image including a road (S710), a step of acquiring image shake information by comparing a first frame (FRAME1) and a second frame (FRAME2) within the image (S720), and a step of correcting the image based on the image shake information (S730).

[0104] Referring to FIG. 8, the image shake information acquisition step (S720) may include a step (S810) of extracting a plurality of first feature points (FP1) from a first frame (FRAME1), a step (S820) of extracting a plurality of second feature points (FP2) from a second frame (FRAME2), a step (S830) of matching a plurality of first feature points (FP1) extracted from the first frame (FRAME1) and a plurality of second feature points (FP2) extracted from the second frame (FRAME2), a step (S840) of calculating movement information for each of a plurality of feature point sets including two feature points (FP1, FP2) matched by a feature point matching unit (322), and a step (S850) of acquiring image shake information based on the movement information.

[0105] Steps S810 to S850 are exemplarily described as follows.

[0106] In step S810, the real-time image stabilization device (120) loads a first frame (FRAME1) of a real-time image and extracts first feature points (FP1) for the loaded first frame (FRAME1). Here, the first frame (FRAME1) is a reference frame for frame comparison.

[0107] For example, the first frame (FRAME1) corresponding to the reference frame may be updated once every several hundred thousand to several million times. In the following description, the case where the first frame (FRAME1) corresponding to the reference frame is updated once every 648,000 frames is taken as an example (6 hours based on 30 fps video: 30 (frames / second) × 60 (seconds / minute) × 60 (minutes / hour) × 6 (hours) = 648,000 (frames)). 648,000 frames and the corresponding time (6 hours) can be referred to as a unit frame set and unit time for real-time video stabilization.

[0108] At step S820, the real-time image stabilization device (120) retrieves a second frame (FRAME2) of the image and extracts second feature points (FP2) from the second frame (FRAME2). Here, the second frame (FRAME2) is a different frame from the first frame (FRAME1), which is a reference frame, and may be a comparison target frame that can be compared with the first frame (FRAME1).

[0109] At step S830, the real-time image stabilization device (120) can compare and match the second feature points (FP2) extracted from the second frame (FRAME2), which is a comparison target frame, with the first feature points (FP1) extracted from the first frame (FRAME1), which is a reference frame.

[0110] After step S830, the real-time image stabilization device (120) can perform video stabilization processing by estimating movement information (movement amount) between feature points by performing steps S840 and S850.

[0111] And, for 6 hours (648,000 frames), the first frame (FRAME1) and the first feature points (FP1) of the first frame (FRAME1) can remain the same. In this sense, the first frame (FRAME1) can be said to be a reference frame, and the first feature points (FP1) can be said to be reference feature points.

[0112] A real-time image stabilization device (120) can maintain the first frame (FRAME1) and the first feature points (FP1) of the first frame (FRAME1) the same for a unit time (e.g., 6 hours) corresponding to a unit frame set (648,000 frames) for real-time image stabilization, continuously change the second frame (FRAME2), which is a frame to be compared, extract the second feature points (FP2), and perform a process of comparing the second feature points (FP2) with the first feature points (FP1), which are reference feature points.

[0113] When a unit time (e.g., 6 hours) corresponding to a unit frame set (648,000 frames) for real-time image stabilization has elapsed, the first frame (FRAME1) corresponding to the reference frame may be changed, and the above-described steps S810 to S850 may be repeated. For example, the first frame (FRAME1) corresponding to the changed reference frame may be a frame included in the following unit frame set (648,000 frames).

[0114] The embodiments of the present disclosure described above are briefly described below.

[0115] A traffic information collection system according to embodiments of the present disclosure may include an image acquisition device that acquires an image including a road, a real-time image stabilization device that acquires image shake information by comparing a first frame and a second frame in the image, and corrects the image based on the image shake information.

[0116] The image acquisition device may be fixed.

[0117] A real-time image stabilization device may include a feature point extraction unit that extracts a plurality of feature points from a first frame and a plurality of feature points from a second frame, a feature point matching unit that matches a plurality of feature points extracted from the first frame with a plurality of feature points extracted from the second frame, a feature point movement information acquisition unit that calculates movement information for each of a plurality of feature point sets including two feature points matched by the feature point matching unit, an image shake information identification unit that identifies image shake information based on the movement information, and an image shake correction unit that corrects an image based on the image shake information.

[0118] The feature point matching unit can match a plurality of feature points extracted from a first frame with a plurality of feature points extracted from a second frame based on the distance between the feature points.

[0119] When a first feature point is extracted from a first frame and a second feature point is extracted from a second frame, if the distance between the first feature point and the second feature point is less than or equal to a threshold value, the first feature point and the second feature point can be matched as feature points for the same point.

[0120] If the distance between the first feature point and the second feature point exceeds the threshold value, the first feature point and the second feature point are not matched as feature points for the same point.

[0121] The image shake information identification unit identifies the image shake size included in the image shake information based on the amount of movement included in the movement information, and the image shake size may be proportional to the amount of movement.

[0122] The image shake information identification unit identifies the image shake direction included in the image shake information based on the movement direction included in the movement information, and the image shake direction can correspond to the movement direction.

[0123] The movement information for each of the plurality of feature point sets can correspond to the movement information of the image acquisition device.

[0124] The movement information for each of the plurality of feature point sets may include at least one of the movement amount and movement direction for each of the plurality of feature point sets.

[0125] The movement information of the image acquisition device may include at least one of the movement amount and movement direction of the image acquisition device.

[0126] The image shake correction unit can correct the image based on a correction direction opposite to the image shake direction included in the image shake information and a correction size corresponding to the image shake size included in the image shake information.

[0127] Correcting an image can mean the same thing as correcting image shake, and it can also mean the same thing as correcting (editing, modifying) an image to remove image shake.

[0128] A plurality of feature points extracted from the first frame and a plurality of feature points extracted from the second frame may correspond to color change points. For example, the color change points may include at least one of a hue change point, a brightness change point, and a saturation change point.

[0129] The plurality of feature points extracted from the first frame and the plurality of feature points extracted from the second frame may correspond to boundary points between a fixed object around the road and the background, or may correspond to boundary points between a first fixed object around the road and a second fixed object.

[0130] A traffic information collection system according to embodiments of the present disclosure may further include a traffic information generation device that generates traffic information on a road based on a corrected image and stores and outputs the traffic information.

[0131] A real-time image stabilization device according to embodiments of the present disclosure may include an input unit for receiving an image, a real-time image stabilization processing unit for obtaining image shake information by comparing a first frame and a second frame in the image, and for correcting the image based on the image shake information.

[0132] The real-time image stabilization processing unit may include a feature point extraction unit that extracts a plurality of feature points from a first frame and a plurality of feature points from a second frame, a feature point matching unit that matches a plurality of feature points extracted from the first frame with a plurality of feature points extracted from the second frame, a feature point movement information acquisition unit that calculates movement information for each of a plurality of feature point sets including two feature points matched by the feature point matching unit, an image shake information identification unit that identifies image shake information based on the movement information, and an image shake correction unit that corrects an image based on the image shake information.

[0133] The feature point matching unit can match a plurality of feature points extracted from a first frame with a plurality of feature points extracted from a second frame based on the distance between the feature points.

[0134] The image shake information identification unit identifies the image shake size included in the image shake information based on the amount of movement included in the movement information, and the image shake size may be proportional to the amount of movement.

[0135] The image shake information identification unit identifies the image shake direction included in the image shake information based on the movement direction included in the movement information, and the image shake direction can correspond to the movement direction.

[0136] Movement information for each of the plurality of feature point sets corresponds to movement information of the image acquisition device, and the movement information for each of the plurality of feature point sets includes at least one of the movement amount and the movement direction for each of the plurality of feature point sets, and the movement information of the image acquisition device may include at least one of the movement amount and the movement direction of the image acquisition device.

[0137] The movement information for each of the plurality of feature point sets may include at least one of vertical component movement information and horizontal component movement information.

[0138] A plurality of feature points extracted from the first frame and a plurality of feature points extracted from the second frame may correspond to color change points.

[0139] A plurality of feature points extracted from the first frame and a plurality of feature points extracted from the second frame may correspond to boundary points between fixed objects around the road and the background.

[0140] A plurality of feature points extracted from the first frame and a plurality of feature points extracted from the second frame can correspond to boundary points between fixed objects around the road.

[0141] A real-time image stabilization method according to embodiments of the present disclosure may include a step of acquiring an image including a road, a step of acquiring image shake information by comparing a first frame and a second frame in the image, and a step of correcting the image based on the image shake information.

[0142] The step of obtaining image shake information may include a step of extracting a plurality of feature points from a first frame, a step of extracting a plurality of feature points from a second frame, a step of matching a plurality of feature points extracted from the first frame with a plurality of feature points extracted from the second frame, a step of calculating movement information for each of a plurality of feature point sets including two feature points matched by a feature point matching unit, and a step of obtaining image shake information based on the movement information.

[0143] According to the embodiments of the present disclosure described above, a traffic information collection system, a real-time image stabilization device, and a real-time image stabilization method that provide a real-time image stabilization function enabling accurate traffic information collection can be provided.

[0144] According to embodiments of the present disclosure, embodiments of the present disclosure can provide a traffic information collection system, a real-time image stabilization device, and a real-time image stabilization method capable of recognizing image shake and performing image correction accordingly.

[0145] The above description merely exemplifies the technical concepts of the present disclosure. Those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present disclosure. Furthermore, the embodiments disclosed in this disclosure are intended to illustrate, rather than limit, the technical concepts of the present disclosure. Therefore, the scope of the technical concepts of the present disclosure is not limited by these embodiments.

[0146]

[0147] CROSS-REFERENCE TO RELATED APPLICATION

[0148] This patent application claims priority under 35 USC §119(a) to Korean Patent Application No. 10-2023-0190226, filed December 22, 2023, the entire contents of which are incorporated herein by reference. Furthermore, this patent application claims priority in countries other than the United States for the same reasons, the entire contents of which are incorporated herein by reference.

Claims

1. An image acquisition device for acquiring an image including a road; and A traffic information collection system including a real-time image stabilization device that obtains image shake information by comparing a first frame and a second frame in the image and corrects the image based on the image shake information.

2. In paragraph 1, The above image acquisition device is a fixed traffic information collection system.

3. In paragraph 1, The above real-time image stabilization device, A feature point extraction unit that extracts a plurality of feature points from the first frame and extracts a plurality of feature points from the second frame; A feature point matching unit that matches a plurality of feature points extracted from the first frame with a plurality of feature points extracted from the second frame; A feature point movement information acquisition unit that calculates movement information for each of a plurality of feature point sets including two feature points matched by the feature point matching unit; An image shake information identification unit that identifies the image shake information based on the above movement information; and A traffic information collection system including an image shake correction unit that corrects the image based on the image shake information.

4. In paragraph 3, A traffic information collection system in which the above-mentioned feature point matching unit matches a plurality of feature points extracted from the first frame with a plurality of feature points extracted from the second frame based on the distance between the feature points.

5. In paragraph 4, When a first feature point is extracted from the first frame and a second feature point is extracted from the second frame, If the distance between the first feature point and the second feature point is less than or equal to a threshold value, the first feature point and the second feature point are matched as feature points for the same point, A traffic information collection system that does not match the first feature point and the second feature point as feature points for the same point if the distance between the first feature point and the second feature point exceeds the threshold value.

6. In paragraph 3, The above image shake information identification unit identifies the image shake size included in the image shake information based on the amount of movement included in the movement information, A traffic information collection system in which the size of the above-mentioned video shake is proportional to the amount of movement.

7. In paragraph 3, The above image shake information identification unit identifies the image shake direction included in the image shake information based on the movement direction included in the movement information, The above video shaking direction corresponds to the above moving direction and is a traffic information collection system.

8. In paragraph 3, A traffic information collection system in which movement information for each of the plurality of feature point sets corresponds to movement information of the image acquisition device and includes at least one of movement information of a vertical component and movement information of a horizontal component.

9. In paragraph 3, The above image shake correction unit is a traffic information collection system that corrects the image based on a correction direction opposite to the image shake direction included in the image shake information and a correction size corresponding to the image shake size included in the image shake information.

10. In paragraph 1, A plurality of feature points extracted from the first frame and a plurality of feature points extracted from the second frame correspond to color change points, A traffic information collection system wherein the above color change points include at least one of a hue change point, a brightness change point, and a saturation change point.

11. In paragraph 1, A plurality of feature points extracted from the first frame and a plurality of feature points extracted from the second frame, Corresponds to boundary points between fixed objects and the background around the above road, or A traffic information collection system corresponding to boundary points between a first fixed object and a second fixed object around the above road.

12. In paragraph 1, A traffic information collection system further comprising a traffic information generation device that generates traffic information on the road based on the corrected image and stores and outputs the traffic information.

13. Input input section for receiving video; and A real-time image stabilization device including a real-time image stabilization processing unit that obtains image shake information by comparing a first frame and a second frame in the image and corrects the image based on the image shake information.

14. In paragraph 13, The above real-time image stabilization processing unit, A feature point extraction unit that extracts a plurality of feature points from the first frame and extracts a plurality of feature points from the second frame; A feature point matching unit that matches a plurality of feature points extracted from the first frame with a plurality of feature points extracted from the second frame; A feature point movement information acquisition unit that calculates movement information for each of a plurality of feature point sets including two feature points matched by the feature point matching unit; An image shake information identification unit that identifies the image shake information based on the above movement information; and A real-time image stabilization device including an image shake correction unit that corrects the image based on the image shake information.

15. In paragraph 14, A real-time image stabilization device in which movement information for each of the above plurality of feature point sets corresponds to the movement information of the image acquisition device.

16. In paragraph 13, A plurality of feature points extracted from the first frame and a plurality of feature points extracted from the second frame correspond to color change points, A real-time image stabilization device wherein the color change points include at least one of a hue change point, a brightness change point, and a saturation change point.

17. Step of acquiring an image including a road; A step of obtaining image shake information by comparing the first frame and the second frame in the above image; and A real-time image stabilization method comprising a step of correcting the image based on the image shake information.

18. In paragraph 17, The step of obtaining the above image shake information is: A step of extracting a plurality of feature points from the first frame; A step of extracting a plurality of feature points from the second frame; A step of matching a plurality of feature points extracted from the first frame with a plurality of feature points extracted from the second frame; A step of calculating movement information for each of a plurality of feature point sets including two feature points matched by the feature point matching unit; and A real-time image stabilization method including a step of obtaining image shake information based on the above movement information.

Images