High-quality image acquisition method and electronic device for performing the same

Abstract

To provide a method of generating an image based on a license plate of a vehicle.SOLUTION: A vehicle license plate image generation method to be performed by an electronic device includes: an operation of generating a plurality of crop images including a license plate area of a target vehicle on the basis of a plurality of basic images of a moving image in which the target vehicle is captured; an operation of determining a reference clop image out of the plurality of crop images; an operation of calculating a plurality of optical flow values between the reference crop image and the plurality of crop images; an operation of determining a plurality of array images out of the plurality of crop images on the basis of the plurality of optical flow values; and an operation of generating a license plate image on the basis of the plurality of array images.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The following embodiments relate to an image generating method and an electronic device for performing the same, and more particularly to a method for generating an image based on a vehicle license plate and an electronic device for performing the same. [Background technology]

[0002] License plate recognition has become an essential technology in many areas of society and various industries, including traffic law compliance and surveillance, security, and crime prevention. However, the performance of license plate recognition models is significantly reduced when using low-quality video.

[0003] To solve this problem, it is necessary to convert low-resolution vehicle license plate images into high-resolution images. However, in this case, it is difficult to fully restore the license plate due to issues such as deterioration of the dashcam and continuous movement of the car and camera.

[0004] It is necessary to provide an image generation method and an electronic device for performing the same that ensures visibility and recognition performance. Summary of the Invention [Problem to be solved by the invention]

[0005] An object of one embodiment is to provide a method for generating an image based on a vehicle license plate.

[0006] An object of one embodiment is to provide an electronic device that generates an image based on a vehicle's license plate.

[0007] However, the technical issues are not limited to those mentioned above, and other technical issues may exist. [Means for solving the problem]

[0008] According to one embodiment, a method for generating a license plate image of a vehicle, which is performed by an electronic device, includes the following operations: generating a plurality of crop images including a license plate area of ​​a target vehicle based on a plurality of base images of a video captured of the target vehicle; determining a reference crop image from the plurality of crop images; calculating a plurality of optical flow values ​​between the reference crop image and each of the plurality of crop images; determining a plurality of aligned images from the plurality of crop images based on the plurality of optical flow values; and generating a license plate image based on the plurality of aligned images.

[0009] According to one embodiment, the operation of generating the plurality of crop images may include an operation of determining the license plate area of ​​the target vehicle in a first base image, an operation of generating a first base crop image including the determined license plate area, and an operation of generating the first crop image by up-sampling the first base crop image.

[0010] According to one embodiment, the reference crop image may be determined to be one of the plurality of crop images based on an image generation time for each of the plurality of crop images.

[0011] According to one embodiment, the operation of calculating the plurality of optical flow values ​​may include an operation of calculating a first position change between a first pixel of the reference crop image and a second pixel of the first crop image corresponding to the first pixel, and an operation of calculating a first optical flow value of the first crop image based on the first position change.

[0012] According to one embodiment, the operation of determining the plurality of aligned images may include an operation of determining an error image among the plurality of crop images based on the plurality of optical flow values, and an operation of determining the plurality of aligned images based on the error image.

[0013] According to one embodiment, the operation of determining the error image may include an operation of determining a first crop image and a second crop image adjacent to the first crop image from among the plurality of crop images, an operation of calculating a difference between a first optical flow value of the first crop image and a second optical flow value of the second crop image, and an operation of determining whether the first crop image is the error image based on the difference.

[0014] According to one embodiment, the operation of generating the license plate image may include an operation of determining a first alignment image that requires correction from the plurality of alignment images, an operation of generating the corrected first alignment image by correcting the first alignment image, and an operation of generating the license plate image based on the plurality of alignment images including the corrected first alignment image.

[0015] According to one embodiment, the operation of determining the first aligned images requiring correction may include an operation of determining whether the license plate region of each of the plurality of aligned images satisfies a bilinear approximation, and an operation of determining the first aligned images that do not satisfy the bilinear approximation as requiring correction.

[0016] According to one embodiment, the operation of generating the corrected first aligned image may include an operation of calculating coordinate values ​​within the first aligned image of corners of the license plate area of ​​the first aligned image, an operation of calculating corrected coordinate values ​​for each of the coordinate values ​​of the corners based on a double linear approximation, and an operation of generating the corrected first aligned image by correcting the first aligned image based on the corrected coordinate values.

[0017] According to one embodiment, the operation of generating the license plate image may include an operation of acquiring intensity values ​​of pixels included in the license plate area of ​​each of the plurality of aligned images, an operation of determining pixels corresponding to a predetermined intensity value from the acquired intensity values, and an operation of generating the license plate image based on the determined pixels.

[0018] According to one embodiment, generating the license plate image may include post-processing the license plate image based on an image enhancement algorithm.

[0019] According to one embodiment, the image improvement algorithm may include a contrast limited adaptive histogram equalization (CLAHE) algorithm.

[0020] According to an embodiment, the method may further include determining the vehicle number of the target vehicle based on the generated license plate image.

[0021] An electronic device according to one embodiment includes at least one processor and a memory for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the electronic device can generate a plurality of cropped images including a license plate area of ​​a target vehicle based on a plurality of base images of a video captured of the target vehicle, determine a reference cropped image from among the plurality of cropped images, calculate a plurality of optical flow values ​​between the reference cropped image and each of the plurality of cropped images, determine a plurality of aligned images from among the plurality of cropped images based on the plurality of optical flow values, and generate a license plate image based on the plurality of aligned images.

[0022] According to one embodiment, when the instruction words are individually or collectively executed by the at least one processor, the electronic device can at least determine the license plate area of ​​the target vehicle in a first base image, generate a first base crop image including the determined license plate area, and generate the first crop image by upsampling the first base crop image.

[0023] According to an embodiment, the reference crop image may be determined to be one of the plurality of crop images based on an image generation time for each of the plurality of crop images.

[0024] According to one embodiment, when the instruction words are individually or collectively executed by the at least one processor, the electronic device can calculate at least a first position change between a first pixel of the reference crop image and a second pixel of the first crop image corresponding to the first pixel, and calculate a first optical flow value of the first crop image based on the first position change.

[0025] According to one embodiment, when the instruction words are executed individually or collectively by the at least one processor, the electronic device can determine an error image among the plurality of cropped images based on at least the plurality of optical flow values, and determine the plurality of aligned images based on the error image.

[0026] According to one embodiment, when the instruction words are executed individually or collectively by the at least one processor, the electronic device can determine at least a first crop image and a second crop image adjacent to the first crop image among the plurality of crop images, calculate a difference between a first optical flow value of the first crop image and a second optical flow value of the second crop image, and determine whether the first crop image is the error image based on the difference.

[0027] According to one embodiment, when the instruction words are executed individually or collectively by the at least one processor, the electronic device can determine at least a first alignment image that requires correction among the plurality of alignment images, generate the corrected first alignment image by correcting the first alignment image, and generate the license plate image based on the plurality of alignment images including the corrected first alignment image.

[0028] According to one embodiment, when the instruction words are executed individually or collectively by the at least one processor, the electronic device can determine at least whether the license plate region of each of the plurality of aligned images satisfies a double linear approximation, and determine that the first aligned image that does not satisfy the double linear approximation requires correction.

[0029] According to one embodiment, when the instruction words are executed individually or collectively by the at least one processor, the electronic device can calculate at least coordinate values ​​within the first aligned image of the corners of the license plate area of ​​the first aligned image, calculate corrected coordinate values ​​for each of the coordinate values ​​of the corners based on double linear approximation, and correct the first aligned image based on the corrected coordinate values ​​to generate the corrected first aligned image.

[0030] According to one embodiment, when the instruction words are individually or collectively executed by the at least one processor, the electronic device can acquire intensity values ​​of pixels included in the license plate area of ​​each of the plurality of aligned images, determine pixels corresponding to a predetermined intensity value from the acquired intensity values, and generate the license plate image based on the determined pixels.

[0031] According to one embodiment, the electronic device is capable of post-processing the license plate image based at least on an image enhancement algorithm when the instructions are individually or collectively executed by the at least one processor.

[0032] According to one embodiment, the image enhancement algorithm may include a CLAHE algorithm.

[0033] According to one embodiment, when the instructions are individually or collectively executed by the at least one processor, the electronic device can determine the vehicle number of the target vehicle based at least on the generated license plate image. [Effects of the Invention]

[0034] According to the present invention, a method for generating an image based on a vehicle license plate can be provided.

[0035] According to the present invention, an electronic device can be provided that generates an image based on a vehicle's license plate. [Brief explanation of the drawings]

[0036] [Figure 1] 1 illustrates an electronic device and a target vehicle according to an example. [Figure 2] FIG. 1 illustrates a system of an electronic device according to an embodiment. [Figure 3] 1 is a flowchart of an image generating method according to an embodiment. [Figure 4] 10 is a flowchart illustrating an operation of generating a cropped image according to an embodiment. [Figure 5] FIG. 1 illustrates a cropped image and a reference cropped image according to an example. [Figure 6] 10 is a flowchart illustrating an operation for calculating an optical flow value according to an embodiment. [Figure 7A] 10A and 10B are diagrams illustrating an operation for calculating an optical flow value according to an example. [Figure 7B] 10A and 10B are diagrams illustrating an operation for calculating an optical flow value according to an example. [Figure 8] 10 is a flowchart illustrating an operation of determining an alignment image according to an embodiment. [Figure 9] 10 is a flowchart illustrating an operation of determining an error image according to an embodiment. [Figure 10] 10 is a flowchart illustrating an operation for generating a license plate image according to an embodiment. [Figure 11] 10 is a flowchart illustrating an operation of determining a first alignment image according to an embodiment. [Figure 12] 10 is a flowchart illustrating an operation of generating a corrected first aligned image according to an embodiment. [Figure 13] 10A and 10B are diagrams illustrating an operation of correcting a first aligned image according to an example. [Figure 14] 10 is a flowchart illustrating an operation for generating a license plate image according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0037] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified in various forms. Therefore, the embodiments are not limited to the specific disclosed forms, and the scope of the present specification includes modifications, equivalents, or alternatives within the technical spirit.

[0038] Although terms such as "first" or "second" may be used to describe multiple components, such terms should be construed only to distinguish one component from the other components. For example, a first component may be designated as a second component, and similarly, a second component may be designated as a first component.

[0039] When any component is referred to as being "coupled" to another component, it should be understood that it may be directly coupled or connected to the other component, but that there may be other components in between.

[0040] The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, the words "comprise" or "have" and the like indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0041] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0042] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. When describing with reference to the drawings, the same reference numerals will be used to designate the same elements, regardless of the reference numerals, and redundant description thereof will be omitted.

[0043] FIG. 1 illustrates an electronic device and a target vehicle according to an example.

[0044] Referring to FIG. 1, an example of a target vehicle 1, a license plate area 12 of the target vehicle, and an electronic device 10 (e.g., a dashcam) are shown. In one example, the electronic device 10 includes a display module, a sensor, and a camera module. In one example, the sensor may be an inertial sensor for measuring changes in acceleration and a speed sensor for determining the speed of the vehicle. The camera module captures images of the surroundings of the vehicle. The electronic device 10 generates one or more frames of the surroundings of the vehicle (e.g., the target vehicle 1) based on the camera module. The electronic device 10 can generate a video of the surroundings of the vehicle based on the one or more frames.

[0045] In one embodiment, the electronic device 10 may generate a video of a target vehicle 1. The electronic device 10 may generate a plurality of cropped images including the license plate area 12 of the target vehicle 1 based on a plurality of base images (e.g., frames) of the video of the target vehicle 1. The cropped images may be portions of the base images including the license plate area 12 of the target vehicle 1. In one embodiment, the electronic device 10 may determine a reference cropped image from the generated plurality of cropped images. The reference cropped image is an image that serves as a reference for the cropped images. In one embodiment, the electronic device 10 may calculate a plurality of optical flow values ​​between the determined reference cropped image and each of the plurality of cropped images. In one embodiment, the electronic device 10 may determine a plurality of alignment images from the plurality of cropped images based on the calculated plurality of optical flow values. The alignment image may be an image from the cropped images that is directly used to generate the license plate image. In one embodiment, the electronic device 10 may generate a license plate image based on the plurality of alignment images. The license plate image may include the license plate area 12 of the target vehicle 1 and may have a higher resolution than the cropped image.

[0046] An image generating method and an electronic device for performing the same will be described in detail below with reference to FIGS.

[0047] FIG. 2 is a diagram illustrating a system of an electronic device according to an embodiment.

[0048] According to one embodiment, electronic device 10 (eg, electronic device 10 of FIG. 1) includes at least one processor 210, a memory 220 for storing instructions, and a communication unit 230.

[0049] The electronic device 10 generates one or more frames of the vehicle's surroundings (e.g., the target vehicle 1 in FIG. 1). The electronic device 10 generates a moving image of the vehicle's surroundings based on the one or more frames.

[0050] When the instruction is executed, the processor 210 may be configured to cause the electronic device 10 to generate a plurality of cropped images including the license plate area of ​​the target vehicle based on a plurality of base images of a video captured of at least the target vehicle, determine a reference cropped image from among the plurality of cropped images, calculate a plurality of optical flow values ​​between the reference cropped image and each of the plurality of cropped images, determine a plurality of aligned images from among the plurality of cropped images based on the plurality of optical flow values, and generate a license plate image based on the plurality of aligned images.

[0051] The processor 210 processes data received by the communication unit 230 and data stored in the memory 220. A "processor" is a data processing device implemented in hardware having circuits with a physical structure for performing desired operations. For example, the desired operations may include code or instructions included in a program. For example, a data processing device implemented in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0052] The processor 210 can execute computer-readable code (eg, software) stored in a memory (eg, memory 220) and instructions issued by the processor 210.

[0053] The memory 220 stores data received by the communication unit 230 and data processed by the processor 210. For example, the memory 220 may store a program (or application, software). The stored program may be coded to control the dashcam 10 and may be a set of syntax executable by the processor 210.

[0054] In one example, memory 220 may include one or more of volatile memory, non-volatile memory, random access memory (RAM), flash memory, a hard disk drive, and an optical disk drive.

[0055] The memory 220 stores an instruction set (e.g., software) that operates the dash cam 10. The instruction set that operates the dash cam 10 may be executed by the processor 210.

[0056] The communication unit 230 is connected to the processor 210 and the memory 220 and can transmit and receive data. The communication unit 230 is connected to other external devices and transmits and receives data. In the following, the expression "transmitting and receiving "A" means transmitting and receiving "information or data indicating A."

[0057] The communication unit 230 can be realized by a circuitry within the dash cam 10. For example, the communication unit 230 may include an internal bus and an external bus. As another example, the communication unit 230 may be an element that connects the dash cam 10 with an external device. The communication unit 230 may be an interface. The communication unit 230 can receive data from an external device and transmit the data to the processor 210 and the memory 220.

[0058] FIG. 3 is a flowchart of an image generating method according to one embodiment.

[0059] The following operations 310-350 may be performed by an electronic device (e.g., the electronic device 10 of FIG. 1 or FIG. 2). For example, the electronic device may include a processor (e.g., the processor 210 of FIG. 2), a memory (e.g., the memory 220 of FIG. 2), and a communication unit (e.g., the communication unit 230 of FIG. 2).

[0060] Before operation 310, the electronic device generates one or more frames of the vehicle's surroundings (e.g., target vehicle 1 in FIG. 1 ). The electronic device can generate a moving image of the vehicle's surroundings based on the one or more frames.

[0061] In operation 310, the electronic device generates multiple crop images including the license plate area of ​​the target vehicle (e.g., license plate area 12 in FIG. 1 ) based on multiple base images of a video captured of the target vehicle. The crop images may be portions of the base images including the license plate area of ​​the target vehicle. The size or resolution of the crop images may vary depending on the distance between the electronic device and the target vehicle. For example, when the distance between the electronic device and the target vehicle is close, the size or resolution of the crop images may be larger than when the distance between the electronic device and the target vehicle is far. The operation of generating multiple crop images will be described in detail below with reference to FIG. 4.

[0062] In operation 320, the electronic device determines a reference crop image from among the plurality of crop images. The reference crop image may be an image to which the crop image is referred. For example, the reference crop image may be determined to be one of the plurality of crop images based on the image generation time for each of the plurality of crop images. The reference crop image will be described in detail below with reference to FIG. 5.

[0063] In operation 330, the electronic device calculates a plurality of optical flow values ​​between the reference crop image and each of the plurality of crop images. The operation of calculating the plurality of optical flow values ​​will be described in detail below with reference to FIGS. 6 to 7B.

[0064] In operation 340, the electronic device determines a plurality of aligned images from the plurality of cropped images based on the plurality of optical flow values. For example, the aligned images may be images from the cropped images that are directly used to generate a license plate image. The operation of determining the plurality of aligned images is described in detail below with reference to FIGS. 8 and 9.

[0065] In operation 350, the electronic device generates a license plate image based on the aligned images. The license plate image includes the license plate area of ​​the target vehicle and has a higher resolution than the cropped image. The operation of generating the license plate image is described in detail below with reference to FIGS. 10 to 14.

[0066] In one embodiment, the electronic device may further perform an operation of determining the license plate number of the target vehicle based on the generated license plate image. For example, the electronic device may recognize or extract the license plate number of the target vehicle as text from the generated license plate image based on an optical character recognition (OCR) method. The method by which the electronic device determines the license plate number of the target vehicle is not limited to the described embodiment.

[0067] FIG. 4 is a flowchart illustrating an operation for generating a cropped image according to an embodiment.

[0068] The following operations 410 to 430 may be performed by an electronic device (e.g., the electronic device 10 of FIG. 1 or FIG. 2). For example, the electronic device includes a processor (e.g., the processor 210 of FIG. 2), a memory (e.g., the memory 220 of FIG. 2), and a communication unit (e.g., the communication unit 230 of FIG. 2).

[0069] According to one embodiment, operation 310 described above with reference to FIG. 3 includes operations 410-430.

[0070] In operation 410, the electronic device determines the license plate area of ​​the target vehicle in the first base image. For example, the first base image may be a full-size image generated by the electronic device. The first base image includes an image of the target vehicle (e.g., target vehicle 1 in FIG. 1 ). For example, the electronic device may determine the license plate area of ​​the target vehicle based on a convolutional neural networks (CNN) method. The method by which the electronic device determines the license plate area of ​​the target vehicle is not limited to the described embodiment.

[0071] In operation 420, the electronic device generates a first basic crop image including the determined license plate area. For example, the first basic crop image may be an image obtained by cropping the remaining area of ​​the first basic image excluding the license plate area of ​​the target vehicle. For example, the size or resolution of the first basic crop image may change depending on the distance between the electronic device and the target vehicle.

[0072] In operation 430, the electronic device generates a first crop image by up-sampling the first base crop image. The first base crop image has a smaller image size or lower resolution when the distance between the electronic device and the target vehicle is large compared to when the distance between the electronic device and the target vehicle is small. Based on the up-sampling method, a crop image with a uniform size can be generated from the base crop images with different sizes. In one example, the sizes of the multiple crop images may be the same.

[0073] FIG. 5 is a diagram illustrating a cropped image and a reference cropped image according to an example.

[0074] Referring to FIG. 5, there is shown a plurality of base images 511, 513, 515, 517, and 519, a plurality of base crop images 531, 533, 535, 537, and 539, and a plurality of crop images 551, 553, 555, 557, and 559 according to one example.

[0075] According to one embodiment, an electronic device (e.g., electronic device 10 of FIG. 1 or FIG. 2) may generate one or more frames including target vehicle 1 prior to operation 310 described above with reference to FIG. 3. The electronic device generates a video of target vehicle 1 based on the one or more frames. The video of target vehicle 1 includes multiple base images 511, 513, 515, 517, and 519. In one example, the size of the image of target vehicle 1 in the multiple base images 511, 513, 515, 517, and 519 may vary based on the distance between the electronic device and target vehicle 1. For example, when the distance between the electronic device and target vehicle 1 is large, the size of the image of target vehicle 1 in the multiple base images 511, 513, 515, 517, and 519 becomes gradually smaller.

[0076] According to one embodiment, the electronic device can generate multiple cropped images 551, 553, 555, 557, 559 including the license plate area 12 of the target vehicle 1 based on multiple base images 511, 513, 515, 517, 519 of a video captured of the target vehicle 1 in operation 310 described above with reference to FIG. 3 .

[0077] Referring again to FIG. 4, in operation 410 described above, the electronic device determines the license plate area 12 of the target vehicle 1 within the plurality of base images 511, 513, 515, 517, 519.

[0078] The electronic device generates a plurality of base crop images 531, 533, 535, 537, and 539 including the determined license plate area 12 in operation 420 described above with reference to Figure 4. In one example, the sizes of the plurality of base crop images 531, 533, 535, 537, and 539 may vary based on the distance between the electronic device and the target vehicle 1. For example, as the distance between the electronic device and the target vehicle 1 increases, the sizes of the plurality of base crop images 531, 533, 535, 537, and 539 become smaller and smaller.

[0079] In operation 430 described above with reference to Figure 4, the electronic device generates multiple crop images 551, 553, 555, 557, and 559 by upsampling multiple base crop images 531, 533, 535, 537, and 539. The multiple crop images 551, 553, 555, 557, and 559 may have the same size. In the process of upsampling the base crop images to the crop images, the multiple crop images 551, 553, 555, 557, and 559 may have different resolutions.

[0080] According to one embodiment, the electronic device may determine a reference crop image from among the plurality of crop images 551, 553, 555, 557, and 559 in operation 320 described above with reference to Figure 3. The reference crop image is an image that is referenced by the crop images 551, 553, 555, 557, and 559. For example, the reference crop image may be determined to be any one of the plurality of crop images 551, 553, 555, 557, and 559 based on the image generation time for each of the plurality of crop images 551, 553, 555, 557, and 559.

[0081] For example, the reference crop image may be determined as the crop image 551 generated at the earliest time among the plurality of crop images 551, 553, 555, 557, and 559 based on the image generation times. For example, the reference crop image may be determined as the crop image 559 generated at the latest time among the plurality of crop images 551, 553, 555, 557, and 559 based on the image generation times. For example, the reference crop image may be determined as the crop image 555 generated at the median time among the plurality of crop images 551, 553, 555, 557, and 559 based on the image generation times. The criteria and method for determining the reference crop image are not limited to the described embodiments.

[0082] For convenience of explanation, the reference crop image will be described below as a crop image 555 generated at an intermediate time among a plurality of crop images 551, 553, 555, 557, and 559 based on the image generation time.

[0083] FIG. 6 is a flowchart illustrating an operation for calculating an optical flow value according to an embodiment.

[0084] The following operations 610 and 620 may be performed by an electronic device (e.g., electronic device 10 of FIG. 1 or FIG. 2). For example, the electronic device includes a processor (e.g., processor 210 of FIG. 2), a memory (e.g., memory 220 of FIG. 2), and a communication unit (e.g., communication unit 230 of FIG. 2).

[0085] According to one embodiment, operation 330, described above with reference to FIG.

[0086] In operation 610, the electronic device calculates a first position change between a first pixel in the reference crop image and a second pixel in the first crop image corresponding to the first pixel. In the multiple crop images, the position and angle of the license plate area of ​​the target vehicle may change over time. Based on the optical flow, a movement pattern between the multiple crop images can be identified. For example, based on the optical flow, the direction or distance of movement of a specific pixel between the first crop image and the second crop image can be calculated. In one example, based on the optical flow, a position change of corresponding pixels in the license plate area of ​​the target vehicle included in each of the multiple crop images can be estimated. In one example, a reference crop image (e.g., reference crop image 555 of FIG. 5) can be used as a reference coordinate for estimating the position change of corresponding pixels. The electronic device can calculate a first position change between a first pixel of the reference crop image 555 (e.g., the upper left corner of the license plate area of ​​the reference crop image) and a second pixel of the first crop image (e.g., the upper left corner of the license plate area of ​​the first crop image) corresponding to the first pixel (e.g., the upper left corner of the license plate area of ​​the first crop image) (e.g., crop image 551 in FIG. 5).

[0087] In operation 620, the electronic device calculates a first optical flow value for the first crop image based on the first position change. In one example, the first optical flow value for the first crop image can be calculated based on a first position change between a first pixel of the reference crop image and a second pixel of the first crop image.

[0088] 7A and 7B are diagrams illustrating an example of an operation for calculating an optical flow value.

[0089] Referring to FIG. 7A, an example reference crop image 555 and multiple crop images 551, 553, 555, 557, and 559 are shown.

[0090] 6, the electronic device calculates a first position change (e.g., a position change between (x3, y3) and (x1, y1)) between a first pixel 715 of the reference crop image 555 and a second pixel 711 of the first crop image 551 that corresponds to the first pixel 715. In this manner, the electronic device can calculate each position change between the first pixel 715 of the reference crop image 555 and each pixel of the multiple crop images that corresponds to the first pixel 715.

[0091] According to one embodiment, the electronic device calculates a first optical flow value for the first cropped image 551 based on the first position change in operation 620 described above with reference to Figure 6. In this manner, the electronic device can calculate optical flow values ​​for each of the multiple cropped images.

[0092] 7B, an example of 30 cropped images is shown. Optical flow values ​​for each of the 30 cropped images are calculated based on operations 610 and 620 described above with reference to FIG. 6. Based on the 30 optical flow values, the cropped images corresponding to frames 01, 24, 25, and 28 are determined to be error images.

[0093] The error image will be described in detail below with reference to FIGS.

[0094] FIG. 8 is a flowchart illustrating an operation of determining an alignment image according to an embodiment.

[0095] The following operations 810 and 820 are performed by an electronic device (e.g., electronic device 10 of FIG. 1 or FIG. 2). For example, the electronic device includes a processor (e.g., processor 210 of FIG. 2), a memory (e.g., memory 220 of FIG. 2), and a communication unit (e.g., communication unit 230 of FIG. 2).

[0096] According to one embodiment, operation 340, described above with reference to FIG.

[0097] In operation 810, the electronic device determines an error image from among the plurality of cropped images based on the plurality of optical flow values. Because the movement of the electronic device and the target vehicle (e.g., target vehicle 1 in FIG. 1) is continuous, the change in the optical flow value between the plurality of cropped images should not be large. For example, if the change in the optical flow value is large, the corresponding cropped image can be estimated as an error image.

[0098] The operation of determining the error image will be described in detail below with reference to FIG.

[0099] In operation 820, the electronic device determines a plurality of alignment images based on the error image. The crop images determined as the error image may be difficult or impossible to correct. In one example, the electronic device may determine the remaining crop images, excluding the error image, from the plurality of crop images as alignment images.

[0100] FIG. 9 is a flowchart illustrating an operation of determining an error image according to an embodiment.

[0101] The following operations 910 to 930 may be performed by an electronic device (e.g., the electronic device 10 of FIG. 1 or FIG. 2). For example, the electronic device includes a processor (e.g., the processor 210 of FIG. 2), a memory (e.g., the memory 220 of FIG. 2), and a communication unit (e.g., the communication unit 230 of FIG. 2).

[0102] According to one embodiment, operation 810 described above with reference to FIG. 8 includes operations 910-930.

[0103] In operation 910, the electronic device determines a first crop image and a second crop image adjacent to the first crop image from among the plurality of crop images. For example, referring to FIG. 7A , the electronic device determines a first crop image 551 and a second crop image 553 adjacent to the first crop image 551 from among the plurality of crop images 551, 553, 555, 557, and 559.

[0104] In operation 920, the electronic device calculates a difference between a first optical flow value of the first crop image and a second optical flow value of the second crop image. For example, referring to FIG. 7A , the first optical flow value of the first crop image 551 can be calculated based on a first position change (e.g., a position change between (x3, y3) and (x1, y1)) between a first pixel 715 of the reference crop image 555 and a second pixel 711 of the first crop image 551 that corresponds to the first pixel 715. For example, referring to FIG. 7A , the second optical flow value of the second crop image 553 can be calculated based on a second position change (e.g., a position change between (x3, y3) and (x2, y2)) between the first pixel 715 of the reference crop image 555 and a third pixel 713 of the second crop image 553 that corresponds to the first pixel 715. In one example, the electronic device can calculate the difference between the first optical flow value of the first crop image 551 and the second optical flow value of the second crop image 553 based on the first position change and the second position change.

[0105] In operation 930, the electronic device determines whether the first crop image is an error image based on the difference. In one example, the electronic device can determine whether the first crop image is an error image based on the difference between adjacent crop images. For example, the electronic device can calculate the absolute value of the difference in optical flow values ​​(e.g., the first optical flow value and the second optical flow value) between adjacent crop images (e.g., the first crop image and the second crop image). The electronic device can calculate the average of the absolute values ​​of the differences in optical flow values ​​between the adjacent crop images. In one example, the electronic device can compare the average of the absolute values ​​with a preset threshold value as shown in Equation (1) below.

[0106]

number

[0107] In formula (1), f t is the optical flow value of the adjacent cropped image at an interval of t from the reference cropped image. For example, θ temp is a preset threshold value. The absolute value of the difference in optical flow values ​​between adjacent cropped images before and after can be compared with the preset threshold value. For example, if all comparison results are greater than the preset threshold value, the corresponding cropped image is determined to be an error image. The calculation method for determining whether a cropped image is an error image is not limited to the described embodiment.

[0108] FIG. 10 is a flowchart illustrating an operation for generating a license plate image according to one embodiment.

[0109] The following operations 1010 to 1030 are performed by an electronic device (e.g., the electronic device 10 of FIG. 1 or FIG. 2). For example, the electronic device includes a processor (e.g., the processor 210 of FIG. 2), a memory (e.g., the memory 220 of FIG. 2), and a communication unit (e.g., the communication unit 230 of FIG. 2).

[0110] According to one embodiment, operations 350 described above with reference to FIG. 3 include operations 1010-1030.

[0111] In operation 1010, the electronic device determines a first alignment image that requires correction from among the plurality of alignment images. The license plate area (e.g., license plate area 12 in FIG. 1) of a target vehicle (e.g., target vehicle 1 in FIG. 1) is planar and adheres to and moves with the target vehicle. For example, based on the optical flow results, the electronic device determines that a non-planar portion of the license plate area requires correction. In one example, the electronic device may determine a first alignment image that corresponds to a non-planar portion from among the plurality of alignment images. The operation of determining a first alignment image that requires correction will be described in detail below with reference to FIG. 11.

[0112] In operation 1020, the electronic device generates a corrected first aligned image by correcting the first aligned image. The operation of generating the corrected first aligned image is described in detail below with reference to FIG.

[0113] At operation 1030, the electronic device generates a license plate image based on the plurality of aligned images, including the corrected first aligned image. In one example, the electronic device can generate the license plate image based on the plurality of conventional aligned images, where no correction is required, and the corrected aligned image.

[0114] FIG. 11 is a flowchart illustrating an operation of determining a first alignment image according to an embodiment.

[0115] The following operations 1110 and 1120 may be performed by an electronic device (e.g., electronic device 10 of FIG. 1 or FIG. 2). For example, the electronic device includes a processor (e.g., processor 210 of FIG. 2), a memory (e.g., memory 220 of FIG. 2), and a communication unit (e.g., communication unit 230 of FIG. 2).

[0116] According to one embodiment, operation 1010 described above with reference to FIG.

[0117] In operation 1110, the electronic device determines whether the license plate region of each of the plurality of aligned images satisfies a bilinear approximation. In one example, the electronic device can determine coordinate values ​​corresponding to four corners (e.g., top, right-hand corner, bottom, and bottom-right corner) of the license plate region of each of the plurality of aligned images. The electronic device can determine whether the four corners of the license plate region of each of the plurality of aligned images satisfies a bilinear approximation.

[0118] In operation 1120, the electronic device determines that the first aligned image that does not satisfy the double linear approximation requires correction. In one example, the electronic device may determine that the aligned image in which the four corners of the license plate area do not satisfy the double linear approximation requires correction.

[0119] FIG. 12 is a flowchart illustrating an operation of generating a corrected first aligned image according to an embodiment.

[0120] The following operations 1210-1230 may be performed by an electronic device (e.g., the electronic device 10 of FIG. 1 or FIG. 2). For example, the electronic device includes a processor (e.g., the processor 210 of FIG. 2), a memory (e.g., the memory 220 of FIG. 2), and a communication unit (e.g., the communication unit 230 of FIG. 2).

[0121] According to one embodiment, operation 1020 described above with reference to FIG. 10 includes operations 1210-1230.

[0122] At operation 1210, the electronic device calculates coordinate values ​​within the first aligned image for corners of the license plate area of ​​the first aligned image. In one example, the electronic device can calculate coordinate values ​​within the first aligned image for corners (e.g., top row, right upper row, bottom row, and bottom right row) of the license plate area of ​​the first aligned image that are determined to require correction.

[0123] In operation 1220, the electronic device calculates corrected coordinate values ​​for each of the corner coordinate values ​​based on the double linear approximation. For example, the electronic device may calculate corrected coordinate values ​​that satisfy the double linear approximation based on the equation of a vertical / horizontal line that passes through the center of the license plate area of ​​the first aligned image. For example, the electronic device may calculate the equation of a vertical / horizontal line that passes through the center of the license plate area of ​​the first aligned image based on a vertical / horizontal median vector. In one example, the electronic device may calculate the equation of the line by excluding values ​​outside a preset range (e.g., values ​​in the top / bottom 15% or values ​​in the top / bottom 7.5%) to reduce the error range. The method for reducing the error range and the preset range are not limited to the described embodiment.

[0124] In act 1230, the electronic device generates a corrected first aligned image by correcting the first aligned image based on the corrected coordinate values. The electronic device may correct the first aligned image determined to require correction based on the corrected coordinate values ​​that satisfy the bilinear approximation. The electronic device may generate a corrected first aligned image by correcting the first aligned image.

[0125] FIG. 13 is a diagram illustrating an operation of correcting a first aligned image according to an example.

[0126] Referring to FIG. 13, a plurality of aligned images 551, 553, 555, and 559 are shown according to one example.

[0127] According to one embodiment, the electronic device determines an error image (e.g., error image 557 in FIG. 7A) from among a plurality of crop images (e.g., a plurality of crop images 551, 553, 555, 557, and 559 in FIG. 7A) in operation 810 described above with reference to FIG. 8. According to one embodiment, the electronic device determines a plurality of aligned images 551, 553, 555, and 559 based on the error image in operation 820 described above with reference to FIG. 8.

[0128] According to one embodiment, the electronic device determines a first aligned image 559 among the plurality of aligned images 551, 553, 555, 559 that requires correction in operation 1010 described above with reference to FIG.

[0129] Referring again to FIG. 11 , in operation 1110 described above, the electronic device determines whether the license plate area of ​​each of the plurality of aligned images 551, 553, 555, and 559 satisfies a double linear approximation. The electronic device may obtain coordinate values ​​corresponding to the four corners of the license plate area of ​​each of the plurality of aligned images 551, 553, 555, and 559. For example, coordinate values ​​1311, 1312, 1313, and 1314 corresponding to the four corners of the license plate area of ​​image 551 may be obtained. The electronic device may determine whether the four corners of each license plate area satisfy a double linear approximation.

[0130] The electronic device determines that the first aligned image that does not satisfy the bilinear approximation requires correction in operation 1120, as described above with reference to Figure 11. For example, the electronic device may determine that the first aligned image 559 that does not satisfy the bilinear approximation requires correction.

[0131] According to one embodiment, the electronic device generates the corrected first aligned image 1359 by correcting the first aligned image 559 that requires correction in operation 1020 described above with reference to FIG.

[0132] Referring again to FIG. 12, in operation 1210 described above, the electronic device calculates the coordinate values ​​within the first aligned image 559 of the corners 1391, 1392, 1393, and 1394 of the license plate area of ​​the first aligned image 559.

[0133] The electronic device calculates corrected coordinate values ​​1395, 1396, 1397, and 1398 for the coordinate values ​​of corners 1391, 1392, 1393, and 1394, respectively, based on the double linear approximation in operation 1220 described above with reference to FIG.

[0134] The electronic device generates corrected first aligned image 1359 by correcting first aligned image 559 based on corrected coordinate values ​​1395, 1396, 1397, and 1398 in operation 1230 described above with reference to FIG.

[0135] According to one embodiment, the electronic device may generate a license plate image based on a plurality of aligned images 551, 553, 555, 1359, including the corrected first aligned image 1359, in operation 1030 described above with reference to FIG.

[0136] FIG. 14 is a flowchart illustrating an operation for generating a license plate image according to one embodiment.

[0137] The following operations 1410-1430 may be performed by an electronic device (e.g., the electronic device 10 of FIG. 1 or FIG. 2). For example, the electronic device includes a processor (e.g., the processor 210 of FIG. 2), a memory (e.g., the memory 220 of FIG. 2), and a communication unit (e.g., the communication unit 230 of FIG. 2).

[0138] According to one embodiment, operations 350 described above with reference to FIG. 3 include operations 1410-1430.

[0139] For example, a license plate area may be divided into a background area and an information area (e.g., a vehicle license plate number such as numbers or text). In the license plate area, the information area may be more important than the background area. According to one example, the electronic device may increase the visibility of the license plate image by differently weighting the background area and the information area.

[0140] At operation 1410, the electronic device obtains intensity values ​​for pixels included in the license plate region of each of the multiple aligned images. For example, colors within an image may be defined for each pixel. Each pixel may have a specific intensity value (or color value). The intensity value of a pixel indicates how light or dark the pixel is, or what color it represents. For example, light colors may have a high intensity, and dark colors may have a low intensity. In one embodiment, the electronic device obtains intensity values ​​for all pixels included in the license plate region of each of the multiple aligned images.

[0141] In operation 1420, the electronic device determines pixels among the acquired intensity values ​​that correspond to a preset intensity value. For example, if the background area is a light-colored index license plate and the information area is a dark-colored index license plate, the electronic device can determine pixels with low intensity values ​​as the information area of ​​the license plate based on the preset intensity value. For example, if the background area is a dark-colored index license plate and the information area is a light-colored index license plate, the electronic device can determine pixels with high intensity values ​​as the information area of ​​the license plate based on the preset intensity value. The preset intensity value varies depending on the color configuration of the license plate and is not limited to the described embodiment.

[0142] In operation 1430, the electronic device generates a license plate image based on the determined pixels. In one example, the electronic device can determine pixels corresponding to information regions based on preset intensity values. The electronic device can generate the license plate image by assigning weights to the pixels corresponding to the information regions.

[0143] According to one embodiment, operation 350 described above with reference to FIG. 3 further includes post-processing the license plate image based on an image enhancement algorithm. In one example, the image enhancement algorithm may include a CLAHE algorithm. Based on the image enhancement algorithm, the visibility of the license plate image is increased. According to one embodiment, the operation of post-processing the license plate image based on the image enhancement algorithm may be performed in parallel with or independently of operations 1410-1430.

[0144] The above-described embodiments may be implemented using hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable array (FPA), programmable logic unit (PLU), microprocessor, or other device that executes and responds to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, a single processing device may be described; however, those skilled in the art will recognize that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0145] Software includes computer programs, codes, instructions, or a combination of one or more thereof, which can configure a processing device to operate as desired or can independently or in combination instruct the processing device. The software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software can be distributed across computer systems coupled to a network and stored and executed in a distributed manner. The software and data can be stored on one or more computer-readable recording media.

[0146] The method according to the present invention may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer-readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and program instructions may be specially designed and constructed for the purposes of the present invention, or may be well-known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language code, such as that generated by a compiler, but also high-level language code that is executed by a computer using an interpreter, for example.

[0147] The hardware devices described above may be configured to operate as one or more software modules to perform the operations described in this invention, and vice versa.

[0148] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, and may be replaced or substituted with other components or equivalents, while still achieving suitable results.

[0149] Accordingly, other implementations, other embodiments, and equivalents of the claims are intended to be within the scope of the following claims.

Claims

1. 1. A method for generating a vehicle license plate image, the method being performed by an electronic device, comprising: generating a plurality of cropped images including a license plate area of ​​the target vehicle based on a plurality of basic images of a video in which the target vehicle is captured; determining a reference crop image from the plurality of crop images; calculating a plurality of optical flow values ​​between the reference crop image and each of the plurality of crop images; determining a plurality of aligned images of the plurality of cropped images based on the plurality of optical flow values; generating a license plate image based on the plurality of aligned images; A method for generating a license plate image of a vehicle, comprising:

2. The operation of generating the plurality of cropped images includes: determining the license plate area of ​​the target vehicle in a first base image; generating a first basic crop image including the determined license plate area; generating a first crop image by up-sampling the first base crop image; 2. The method of claim 1, comprising:

3. The method of claim 1 , wherein the reference crop image is determined to be one of the plurality of crop images based on an image generation time for each of the plurality of crop images.

4. The operation of calculating the plurality of optical flow values ​​includes: calculating a first positional change between a first pixel of the reference crop image and a second pixel of the first crop image corresponding to the first pixel; calculating a first optical flow value of the first cropped image based on the first position change; 2. The method of claim 1, comprising:

5. The operation of determining the plurality of aligned images includes: determining an error image from the plurality of cropped images based on the plurality of optical flow values; determining the plurality of aligned images based on the error image; 2. The method of claim 1, comprising:

6. The operation of determining the error image comprises: determining a first crop image and a second crop image adjacent to the first crop image from among the plurality of crop images; calculating a difference between a first optical flow value of the first cropped image and a second optical flow value of the second cropped image; determining whether the first cropped image is the error image based on the difference; 6. The method of claim 5, further comprising:

7. The operation of generating the license plate image includes: determining a first alignment image that requires correction from among the plurality of alignment images; correcting the first-aligned image to generate a corrected first-aligned image; generating the license plate image based on the plurality of aligned images including the corrected first aligned image; 2. The method of claim 1, comprising:

8. The operation of determining the first aligned image requiring correction includes: determining whether the license plate region of each of the plurality of aligned images satisfies a bilinear approximation; determining that the first aligned image does not satisfy the bilinear approximation as requiring correction; 8. The method of claim 7, comprising:

9. The act of generating the corrected first aligned image comprises: calculating coordinate values ​​within the first aligned image of corners of the license plate area of ​​the first aligned image; calculating corrected coordinate values ​​for each of the coordinate values ​​of the corners based on a double linear approximation; generating the corrected first-aligned image by correcting the first-aligned image based on the corrected coordinate values; 8. The method of claim 7, comprising:

10. The operation of generating the license plate image includes: obtaining intensity values ​​of pixels included in the license plate region of each of the plurality of aligned images; determining pixels corresponding to a predetermined intensity value among the acquired intensity values; generating the license plate image based on the determined pixels; 2. The method of claim 1, comprising:

11. 2. The method of claim 1, wherein generating the license plate image comprises post-processing the license plate image based on an image enhancement algorithm.

12. 12. The method of claim 11, wherein the image improvement algorithm comprises a contrast limited adaptive histogram equalization (CLAHE) algorithm.

13. The method of claim 1 , further comprising the operation of determining the vehicle number of the target vehicle based on the generated license plate image.

14. A computer program stored on a computer-readable recording medium for executing the method according to any one of claims 1 to 13 in combination with hardware.

15. 1. An electronic device, comprising: at least one processor; a memory for storing instructions; Including, The instructions, when executed individually or collectively by the at least one processor, cause the electronic device to perform at least: generating a plurality of cropped images including a license plate area of ​​the target vehicle based on a plurality of basic images of a video in which the target vehicle is captured; determining a reference crop image from the plurality of crop images; calculating a plurality of optical flow values ​​between the reference cropped image and each of the plurality of cropped images; determining a plurality of aligned images of the plurality of cropped images based on the plurality of optical flow values; an electronic device that generates a license plate image based on the plurality of aligned images.

Images