System, device, server, method and computer program for compensating image

The image correction system enhances inspection reliability and accuracy by aligning input images with reference images, addressing camera viewpoint changes and environmental distortions, and optimizing resource use.

KR103043942B1Active Publication Date: 2026-11-10세이지 주식회사
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
KR1020250080884
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2025-03-06
Filing Date
2025-06-19
Publication Date
2026-11-10
Estimated Expiration
2045-06-19

AI Technical Summary

Technical Problem

Existing image correction systems struggle with detecting changes in camera viewpoint and environmental distortions, leading to reduced inspection reliability and accuracy in product quality assessment, while being dependent on external equipment.

Method used

An image correction system that generates template images from reference and input images, compares them to detect errors, and corrects the input image to align with the reference, activating correction only when necessary, thus minimizing resource consumption and environmental adaptability.

Benefits of technology

Improves inspection reliability and accuracy by aligning input images with reference images, reducing system processing speed, and minimizing dependency on external equipment, while ensuring robustness to environmental changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112025068675081-PAT00002_ABST
    Figure 112025068675081-PAT00002_ABST
Patent Text Reader

Abstract

A system according to one embodiment of the present disclosure comprises, in a system for correcting an input image, an image comparison unit that generates a first template image based on first region of interest information including information about a reference image and a first region of interest, generates a second template image based on an input image, and generates a trigger signal based on the first template image and the second template image, and an image correction unit that compares the reference image and the input image and corrects the input image based on the comparison result to generate a corrected image, wherein the image correction unit is activated or deactivated according to the trigger signal.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present disclosure relates to an image correction system, device, server, method, and computer program, and more specifically, to an image correction system, device, server, method, and computer program capable of detecting image errors and performing image correction based thereon. Background Technology

[0002] Recently, interest in technologies such as artificial intelligence, big data, the Internet of Things, and cloud computing has been on the rise. Here, Artificial Intelligence (AI) refers to technology capable of reasoning, learning, and acting using data that requires human intelligence or is larger in scale than humans can analyze. With recent improvements in computing efficiency, AI-based computer vision technology is rapidly advancing.

[0003] Computer vision technology is a technology that extracts meaningful information from images, videos, and other visual inputs using computers and systems, and then performs tasks or makes recommendations based on this information.

[0004] Meanwhile, product manufacturing requires inspection to determine whether any defects exist in the products. Recently, efforts to reduce costs, such as the automation of production lines, have been underway, and consequently, interest in automating product quality inspection is also increasing. The problem to be solved

[0005] One objective of the present disclosure is to provide an image correction system, device, server, method, and computer program capable of detecting image errors such as changes in camera viewpoint.

[0006] Another objective of the present disclosure is to provide an image correction system, device, server, method, and computer program capable of improving inspection reliability and inspection accuracy.

[0007] Another objective of the present disclosure is to provide an image correction system, device, server, method, and computer program with improved efficiency.

[0008] Another objective of the present disclosure is to provide an image correction system, device, server, method, and computer program that are robust to changes in the environment of the scene where the image is captured.

[0009] Another objective of the present disclosure is to provide an image correction system, device, server, method, and computer program that can minimize dependency on external equipment.

[0010] The purposes of this disclosure are not limited to those mentioned above, and other unmentioned purposes will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0011] In one embodiment of the present disclosure, the system comprises, in a system for correcting an input image, an image comparison unit that generates a first template image based on first region of interest information including information about a reference image and a first region of interest, generates a second template image based on the input image, and generates a trigger signal based on the first template image and the second template image, and an image correction unit that compares the reference image and the input image and corrects the input image based on the comparison result to generate a corrected image, wherein the image correction unit may be activated or deactivated according to the trigger signal.

[0012] In one embodiment of the present disclosure, the device comprises, in a device for correcting an input image, a memory for storing at least one instruction, and a processor for performing at least one process for correcting the input image in response to the instruction stored on the memory, wherein the processor may generate a first template image based on first region of interest information including information about a reference image and a first region of interest, generate a second template image based on the input image, and generate a trigger signal based on the first template image and the second template image. Depending on the trigger signal, the processor may i) compare the reference image and the input image and correct the input image based on the comparison result to generate and output a corrected image, or ii) output the input image as the corrected image.

[0013] In one embodiment of the present disclosure, a computer program is a computer program stored on a computer-readable medium, and when executed by a computer system, the computer system may include instructions for performing a method comprising: generating a first template image based on first region of interest information including information about a reference image and a first region of interest; generating a second template image based on an input image; generating a trigger signal based on the first template image and the second template image; and outputting a correction image. The step of outputting the correction image may, depending on the trigger signal, i) compare the reference image and the input image and correct the input image based on the comparison result to generate and output the correction image, or ii) output the input image as the correction image.

[0014] Specific details of other embodiments are included in the description of the invention and the drawings. Effects of the invention

[0015] An image correction system, device, server, method, and computer program according to embodiments of the present disclosure can detect whether an image error, such as a change in camera viewpoint or camera distortion, has occurred by comparing an input image and a reference image.

[0016] In addition, the image correction system, device, server, method, and computer program according to the embodiments of the present disclosure correct the input image to match (or align) the reference image using geometric-based alignment when an image error occurs, thereby enabling more precise correction of the input image. Accordingly, since the input image used for product appearance inspection can be corrected to correspond to the viewpoint of the reference image regardless of changes or distortions in the camera viewpoint, the reliability and accuracy of the inspection can be improved during product appearance inspection using the image.

[0017] Herein, an image correction system, device, server, method, and computer program according to embodiments of the present disclosure correct an input image to match (or align) a reference image only when an image error occurs (or when the image error exceeds a preset range), and can deactivate a module for image correction (e.g., an image correction unit) when no image error occurs (or when the image error is within a preset range). That is, since a subsequent module for image correction (e.g., an image correction unit) is activated and operates only when correction is required, a decrease in system processing speed and unnecessary resource consumption can be prevented, and accordingly, system operational efficiency can be improved.

[0018] In addition, when using an image correction system, device, server, method, and computer program according to the embodiments of the present disclosure, hardware equipment required in the prior art is not separately required, so adaptability to changes in the image shooting environment is ensured, robustness to environmental changes is possible, and dependence on external equipment for correcting input images can be minimized.

[0019] However, the effects of the present disclosure are not limited to those exemplified above, and various other effects are included within the present disclosure and may be extended in various ways without departing from the spirit and scope of the present disclosure. Brief explanation of the drawing

[0020] FIG. 1 is a block diagram showing an electronic device according to various embodiments of the present disclosure. FIG. 2 is a block diagram showing an image correction system according to various embodiments of the present disclosure. FIGS. 3A and FIGS. 3B are block diagrams showing an example of a corrector included in the image correction system of FIG. 2. FIG. 4 is a block diagram showing an example of an image comparison unit included in the corrector of FIG. 3a. FIG. 5 is a diagram illustrating an example of the operation of a first template image generation unit included in the image comparison unit of FIG. 4. FIGS. 6a and FIGS. 6b are drawings for explaining an example of the operation of a second template image generation unit included in the image comparison unit of FIGS. 4. FIG. 7 is a diagram illustrating another example of the operation of the first template image generation unit and the second template image generation unit included in the image comparison unit of FIG. 4. FIG. 8 is a block diagram showing an example of an image correction unit included in the corrector of FIG. 3a. FIG. 9 is a flowchart illustrating an image correction method according to various embodiments of the present disclosure. Specific details for implementing the invention

[0021] In describing each drawing, similar reference numerals have been used for similar components. In the attached drawings, the dimensions of the structures are depicted enlarged from their actual size for clarity of the present disclosure. The area and thickness of each component shown in the drawings are depicted for convenience of explanation and are not necessarily limited to the area and thickness of the components depicted in the present disclosure.

[0022] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0023] In this disclosure, terms such as “comprising,” “having,” and “consisting of” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should not be understood as precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0024] The features of each of the various embodiments of the present disclosure may be combined or combined with one another in part or wholly, and various technical interlocking and operation may be possible, and each embodiment may be implemented independently of one another or together in an interlocking relationship.

[0025] Hereinafter, various embodiments of the present disclosure will be described in more detail with reference to the attached drawings. Identical reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0027] <본 개시의 실시예들에 따른 장치>

[0028] FIG. 1 is a block diagram showing an electronic device (100) according to various embodiments of the present disclosure.

[0029] Referring to FIG. 1, an electronic device (100) according to embodiments of the present disclosure may include a processor (110), a communication module (120), an input module (130), a display module (140), and a memory (150).

[0030] According to an embodiment, a processor (110), a communication module (120), an input module (130), a display module (140), and a memory (150) included in an electronic device (100) may each be electrically and / or physically connected to each other. Meanwhile, although FIG. 1 is illustrated as having an electronic device (100) that includes a processor (110), a communication module (120), an input module (130), a display module (140), and a memory (150), this is merely illustrative and the embodiments of the present disclosure are not limited thereto. For example, the electronic device (100) may be implemented with some of the above-described components omitted or with additional components other than those described above (e.g., a power management module, a battery, etc.).

[0031] The processor (110) can control at least one other component (e.g., a hardware or software component) of the electronic device (100) connected to the processor (110) by executing software (e.g., a program (151)), and can perform various data processing or operations. In one embodiment, as at least part of the data processing or operations, the processor (110) can store commands or data received from other components (e.g., an input module (130) or a communication module (120)) in volatile memory, process the commands or data stored in volatile memory, and store the resulting data in non-volatile memory. For example, memory (150) may include volatile memory or non-volatile memory.

[0032] In one embodiment, the processor (110) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (100) includes a main processor and an auxiliary processor, the auxiliary processor may be configured to use less power than the main processor or to be specialized for a designated function. Depending on the embodiment, the auxiliary processor may be implemented separately from the main processor or as part thereof.

[0033] The processor (110) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence model may be generated through machine learning. Machine learning refers to learning that optimizes parameters with given data using a model composed of multiple parameters. Such learning may be performed, for example, on the electronic device (100) itself where the artificial intelligence is performed, or through a separate server. An artificial intelligence learning paradigm refers to various learning methods and approaches, and may include, for example, supervised learning, unsupervised learning, semisupervised learning, reinforcement learning, self-supervised learning, or transfer learning. An artificial intelligence learning algorithm may include, for example, linear regression, logistic regression, decision trees, random forest, support vector machines (SVM), or artificial neural networks, but is not limited to the examples described above.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a deep convolutional neural network (DCNN), a recurrent neural network (RNN), a generative adversarial network (GAN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, a feedforward algorithm, or a combination of two or more of the above, but is not limited to the examples described above. An artificial intelligence model may include multiple artificial neural network layers. In addition to a hardware structure, an artificial intelligence model may additionally or substantially include a software structure.

[0034] The communication module (120) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (100) and an external electronic device (e.g., an electronic device or server other than the electronic device (100)), and the performance of communication through the established communication channel. The communication module (120) may include one or more communication processors that operate independently of the processor (110) and support direct communication (e.g., wired communication) or wireless communication. In one embodiment, the communication module (120) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (e.g., a LAN (local area network) communication module, or a power line communication module). The corresponding communication module among these communication modules is Bluetooth TM It can communicate with external electronic devices through local area communication networks such as Wi-Fi (wireless fidelity) direct or IrDA (infrared data association), legacy cellular networks, 5G networks, next-generation communication networks, the Internet, or computer networks (e.g., LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0035] The input module (130) may receive commands or data to be used for a component of the electronic device (100) (e.g., processor (110)) from outside the electronic device (100) (e.g., user). The input module (130) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen), but is not limited thereto.

[0036] The display module (140) can visually provide information to an external (e.g., user) of the electronic device (100). The display module (140) may include, for example, a display (e.g., a display panel), a holographic device, or a projector, and may further include a control circuit for controlling said device. Additionally, the display module (140) may further include a touch screen to recognize touch input from the user.

[0037] In one embodiment, the display module (140) can display a graphic user interface (GUI) for receiving user input (e.g., touch input, etc.).

[0038] The memory (150) can store various data used by at least one component of the electronic device (100) (e.g., processor (110) or input module (130)). The data may include, for example, input data or output data for software (e.g., program (151)) and related instructions.

[0039] In one embodiment, memory (150) may store a learned model (e.g., a model learned through a processor (110) or a separate server). For example, at least some of the programs (151) stored in memory (150) may be learned models. Accordingly, the processor (110) may use the learned models by executing the programs (151).

[0040] The program (151) may be stored as software in memory (150) and may include, for example, an operating system, middleware, or an application.

[0042] <본 개시의 실시예들에 따른 시스템>

[0043] FIG. 2 is a block diagram showing an image correction system (200) according to various embodiments of the present disclosure.

[0044] Referring to FIG. 2, an image correction system (200) according to embodiments of the present disclosure may include a learner (210) and a corrector (220).

[0045] The learning device (210) is configured to learn an artificial intelligence model, and the learning device (210) can learn at least one artificial intelligence model necessary for the image correction system (200) to correct an input image and generate a corrected image.

[0046] In one embodiment, the learner (210) may learn an artificial intelligence model using at least one of the artificial intelligence learning algorithms described with reference to FIG. 1 and / or at least one of the artificial neural networks, but is not limited thereto.

[0047] The corrector (220) can generate a corrected image by calibrating an input image using at least one artificial intelligence model learned by the learner (210). For example, the corrector (220) can generate a corrected image by comparing the input image with a reference image to detect whether an image error, such as a change in camera viewpoint or camera distortion, has occurred, and if an image error has occurred, it can generate a corrected image by calibrating the input image to match (or align) the reference image. However, the embodiments of the present disclosure are not limited thereto, and the corrector (220) may use at least one pre-learned artificial intelligence model.

[0048] According to an embodiment, an image correction system (200) according to the embodiments of the present disclosure can correct an image used for product inspection. For example, in order to perform an external inspection of a product (e.g., external defect inspection, etc.) using an image acquired from a camera (e.g., CCTV, etc.) in an environment such as a product production line, it is necessary for the image (or image segment) of the product to be inspected to be included in the image, and for precise inspection, it is necessary to photograph the product from a fixed position without any change in the viewpoint or movement of the camera capturing the product. However, in a product production site, the position or angle of the camera may change slightly over time. In this case, the position and shape of the product in the image acquired by the camera may differ from the reference image, and this becomes a major factor affecting the accuracy of the product inspection. In other words, if a situation such as a change in the camera's viewpoint or camera tilt occurs and an external inspection of the product is performed using an image acquired from the camera, the inspection accuracy may be significantly reduced.

[0049] Accordingly, the image correction system (200) according to the embodiments of the present disclosure, for example, the corrector (220), can detect whether an image error, such as a change in camera viewpoint or camera distortion, has occurred by comparing an input image and a reference image. For example, the reference image may be a previously stored image data obtained by capturing a preset location, and the corrector (220) can calculate a similarity score between the input image and the reference image by comparing at least a portion of the input image with at least a portion of the reference image, and based on this, detect (or determine) whether an image error has occurred due to a change in camera viewpoint or camera distortion. In addition, if an image error has occurred, the corrector (220) can generate a corrected image by correcting the input image to match (or align) the reference image. In this way, the image correction system (200) (e.g., corrector (220)) according to the embodiments of the present disclosure corrects an image (input image) used for an appearance inspection of a product (e.g., appearance defect inspection, etc.), thereby improving the reliability and accuracy of the inspection when inspecting the appearance of a product through the image.

[0050] In one embodiment, the corrector (220) can determine whether to activate a subsequent module for image correction (e.g., the image correction unit of FIG. 3a) depending on whether an image error occurs. For example, if no image error occurs in the input image or if the degree of image error is within a preset range, there is no need to separately correct and output the input image; therefore, the corrector (220) can activate the subsequent module for image correction (e.g., the image correction unit of FIG. 3a) only when image correction is required, depending on whether an image error occurs. That is, since the subsequent module for image correction (e.g., the image correction unit of FIG. 3a) is activated and operates only when correction is required, the system operation efficiency of the image correction system (200) (e.g., the corrector (220)) can be improved.

[0051] According to an embodiment, the image correction system (200) may include additional configurations in addition to the learner (210) and the corrector (220). For example, the image correction system (200) may further include a communication module (e.g., the communication module (120) of FIG. 1), an input module (e.g., the input module (130) of FIG. 1), a display module (e.g., the display module (140) of FIG. 1), or a memory (e.g., the memory (150) of FIG. 1). In one example, a model learned through the learner (210) may be stored in the memory of the image correction system (200) (e.g., the memory (150) of FIG. 1).

[0052] In one embodiment, with reference to FIG. 1 and FIG. 2, the image correction system (200) of FIG. 2 may be the electronic device (100) described with reference to FIG. 1 and / or may be implemented as the electronic device (100).

[0053] For example, the operation of the learner (210) and corrector (220) shown in FIG. 2 can be performed by the processor (110) shown in FIG. 1. For example, the image correction system (200) includes a processor (110) (e.g., CPU), and the processor (110) of the image correction system (200) can use an artificial intelligence model, for example, a learned model stored in memory (150), to compare an input image input to the image correction system (200) with a reference image to detect whether an image error, such as a change in camera viewpoint or camera distortion, has occurred, and if an image error has occurred, the input image can be corrected to match (or be aligned with) the reference image to generate a corrected image.

[0054] As another example, the learner (210) and corrector (220) illustrated in FIG. 2 may be at least one processor included in the processor (110) illustrated in FIG. 1 and / or may be performed by said at least one processor. For example, the learner (210) and corrector (220) may be at least one auxiliary processor (e.g., GPU) included in the processor (110) and / or may be performed by said at least one auxiliary processor.

[0055] Hereinafter, with reference to FIGS. 3a to 8, the operation of an image correction system (200) (or electronic device (100)) according to embodiments of the present disclosure will be described in more detail.

[0057] <보정기의 실시예>

[0058] FIGS. 3a and FIGS. 3b are block diagrams showing an example of a corrector (200, 200_1) included in the image correction system (200) of FIG. 2.

[0059] Referring to FIGS. 2 and FIGS. 3a, the corrector (220) can calculate a similarity score between the input image (INPUT) and the reference image (REF) by comparing at least a portion of the input image (INPUT) with at least a portion of the reference image (REF), and can detect (or determine) whether an image error has occurred due to a change in the camera viewpoint or camera distortion, etc. based on the result of comparing the similarity score with a threshold value. In addition, the corrector (220) can generate a trigger signal (TRG) to activate a subsequent module (e.g., image correction unit (222)) when an image error occurs, and can generate a corrected image (OUT) by using the subsequent module (e.g., image correction unit (222)) to correct the input image (INPUT) so that it is matched (or aligned) with the reference image (REF).

[0060] To this end, in one embodiment, the corrector (220) may include an image comparison unit (221) and an image correction unit (222).

[0061] The image comparison unit (221) can receive information about the input image (INPUT), reference image (REF), and the first region of interest (ROI1) (hereinafter referred to as "first region of interest information").

[0062] According to an embodiment, the input image (INPUT) may be a still image. For example, the input image (INPUT) may correspond to any one selected frame image among a plurality of frames included in a video acquired from a camera (e.g., CCTV). As an example, a single frame image that is most suitable (optimal) for product appearance inspection among a plurality of frames included in a video acquired by filming a product production line using a camera (e.g., CCTV) for product appearance inspection (e.g., appearance defect inspection, etc.) may be selected and input as the input image (INPUT).

[0063] However, the embodiments of the present disclosure are not limited thereto, and a still image may be obtained from a camera (e.g., CCTV) and said still image may be input as an input image (INPUT).

[0064] In another embodiment, the input image (INPUT) may be a video. In this case, the image comparison unit (221) may perform the operation of the image comparison unit (221) described below for each of the plurality of frames included in the input image (INPUT) which is a video (e.g., calculating a similarity score through comparison with a reference image (REF), and detecting whether an image error has occurred based on the result of comparing the similarity score with a threshold value).

[0065] For the sake of convenience, the following explanation will be based on the assumption that the input image is a still image.

[0066] The reference image (REF) is a still image and may be a still image obtained by capturing a preset position. For example, the reference image (REF) may be a still image obtained by capturing a position optimized for product appearance inspection on a product production line and may be stored image data.

[0067] The first region of interest information (or, first region of interest data) may include information about a region of interest (ROI) within a reference image (REF) (hereinafter referred to as "first region of interest (ROI1)"). For example, the first region of interest (ROI1) may be set as at least a portion of the reference image (REF) so that the image comparison unit (221) determines whether the input image (INPUT) and the reference image (REF) are matched.

[0068] According to the embodiment, the first region of interest information for the reference image (REF) and the first region of interest (ROI1) may be pre-stored data. For example, referring further to FIG. 3b, the corrector (220_1) may further include a memory (223). The memory (223) includes the first region of interest information for the reference image (REF) and the first region of interest (ROI1), and the reference image (REF) is provided from the memory (223) to the image comparison unit (221) and the image correction unit (222), and the first region of interest information may be provided to the image comparison unit (221).

[0069] Referring again to FIG. 3a, the image comparison unit (221) can detect (or determine) whether an image error has occurred based on the input image (INPUT), reference image (REF), and first region of interest information.

[0070] In one embodiment, the image comparison unit (221) may generate a second region of interest based on a first region of interest. For example, the image comparison unit (221) may generate a second region of interest by expanding the first region of interest, but is not limited thereto.

[0071] Additionally, the image comparison unit (221) may extract a template image (or image segment) of an area overlapping with a first region of interest within a reference image (REF), and extract a template image (or image segment) of an area overlapping with a second region of interest within an input image (INPUT). For convenience of explanation, the template image of an area overlapping with a first region of interest within the reference image (REF) described above may be named "first template image," and the template image of an area overlapping with a second region of interest within an input image (INPUT) may be named "second template image."

[0072] In one embodiment, the image comparison unit (221) can calculate a similarity score between an input image (INPUT) and a reference image (REF) based on a first template image and a second template image. For example, the image comparison unit (221) can use a template matching technique to search for matching locations between the first template image and the second template image, and calculate (or extract, generate) a similarity score based thereon.

[0073] Additionally, the image comparison unit (221) can detect (or determine) whether an image error has occurred based on the calculated similarity score. For example, if the similarity score is less than a preset threshold, the image comparison unit (221) can determine that an image error has occurred due to a change in the camera's viewpoint or camera distortion, etc. (or determine that the image error has exceeded a preset range), and if the similarity score is greater than or equal to the preset threshold, it can determine that no image error has occurred (or determine that the image error is within a preset range).

[0074] In one embodiment, if the image comparison unit (221) determines that an image error has occurred, it determines that correction of the image is necessary and can activate the subsequent module, the image correction unit (222). For example, the image comparison unit (221) may be a preprocessing judgment module for the subsequent module, the image correction unit (222).

[0075] For example, if the image comparison unit (221) determines that an image error has occurred, it may generate and / or output a trigger signal (TRG) to activate the image correction unit (222). In this case, the trigger signal (TRG) may be provided to the image correction unit (222) to activate the image correction unit (222). For example, when the image correction unit (222) is activated, the image correction unit (222) may correct the input image (INPUT) to generate and / or output a corrected image (OUT).

[0076] In contrast, if the image comparison unit (221) determines that no image error has occurred, it may not generate (or output) a trigger signal (TRG). In this case, since the trigger signal (TRG) is not provided to the image correction unit (222), the image correction unit (222) may be deactivated. For example, when the image correction unit (222) is deactivated, the image correction unit (222) may output the input image (INPUT) as a corrected image (OUT) without any separate operation (driving) (e.g., bypass), or the input image (INPUT) input to the image correction system (200), e.g., the corrector (220), may be output as a corrected image (OUT) without passing through the image comparison unit (221).

[0077] In this way, the image correction unit (222) can be activated or deactivated according to the trigger signal (TRG).

[0078] Meanwhile, although the image comparison unit (221) has been described above as generating (or outputting) a trigger signal (TRG) when it is determined that an image error has occurred and generating (or outputting) a trigger signal (TRG) when it is determined that no image error has occurred, the embodiments of the present disclosure are not limited thereto. For example, the image comparison unit (221) may generate (or output) a trigger signal (TRG) that instructs activation (e.g., a trigger signal (TRG) having a value of 1) when it is determined that an image error has occurred, and generate (or output) a trigger signal (TRG) that instructs deactivation (e.g., a trigger signal (TRG) having a value of 0) when it is determined that no image error has occurred.

[0079] The image correction unit (222) can receive an input image (INPUT), a reference image (REF), and a trigger signal (TRG). When the image correction unit (222) is activated based on the trigger signal (TRG), the image correction unit (222) can correct the input image (INPUT) to match (or align) the reference image (REF) and generate a corrected image (OUT). For example, the image correction unit (222) can quantitatively analyze the viewpoint difference between the reference image (REF) and the input image (INPUT) and correct the input image (INPUT) to match (or align) the reference image (REF) through geometric transformation.

[0080] In one embodiment, the image correction unit (222) can correct the input image (INPUT) to match the reference image (REF) by utilizing an image registration technique. For example, the image correction unit (222) can calculate a perspective transform by estimating a correspondence relationship based on corresponding points (feature points) between the input image (INPUT) and the reference image (REF), and use this to align the input image (INPUT) to the reference image (REF). As an example, the image correction unit (222) may use a homography-based registration technique, but is not limited thereto.

[0081] For example, the image correction unit (222) can extract a first feature point from a reference image (REF), extract a second feature point from an input image (INPUT), and form a match pair by matching the first feature point and the second feature point. Additionally, the image correction unit (222) can use this to calculate (or calculate, generate) a homography matrix, and use the calculated homography matrix to transform or correct the input image (INPUT) through a warp transform so that the input image (INPUT) is aligned with the reference image (REF).

[0082] However, the embodiments of the present disclosure are not limited thereto, and the image correction unit (222) may use various techniques and / or artificial intelligence models to correct the input image (INPUT) so that it is matched (or aligned) with the reference image (REF).

[0083] In the following, regarding the operation of the corrector (220, 220_1), the operation of the image comparison unit (221) will be explained in more detail with reference to FIGS. 4 to 7, and the operation of the image correction unit (222) will be explained in more detail with reference to FIG. 8.

[0085] <영상 비교부의 실시예>

[0086] FIG. 4 is a block diagram showing an example of an image comparison unit (221) included in the corrector (220) of FIG. 3a.

[0087] FIG. 5 is a diagram for explaining an example of the operation of a first template image generation unit (2212) included in the image comparison unit (221) of FIG. 4.

[0088] FIGS. 6a and FIGS. 6b are drawings for explaining an example of the operation of a second template image generation unit (2213) included in the image comparison unit (221) of FIG. 4.

[0089] FIG. 7 is a drawing for explaining another example of the operation of the first template image generation unit (2212) and the second template image generation unit (2213) included in the image comparison unit (221) of FIG. 4.

[0090] For example, FIG. 5 illustrates a reference image (REF), a first region of interest (ROI1) within the reference image (REF), and a first template image (TPIG1) generated by the first template image generation unit (2212) based on the reference image (REF) and the first region of interest (ROI1) to explain the operation of the first template image generation unit (221) included in the image comparison unit (221).

[0091] Additionally, FIGS. 6a and 6b illustrate an input image (INPUTa, INPUTb), a second region of interest (ROI2a, ROI2b) within the input image (INPUTa, INPUTb), and a second template image (TPIG2a, TPIG2b) generated by the second template image generation unit (2213) based on the input image (INPUTa, INPUTb) and the second region of interest (ROI2a, ROI2b) in order to explain the operation of the second template image generation unit (221).

[0092] In addition, FIG. 7 shows a reference image (REF) and an input image (INPUT), along with a first region of interest (ROI1_1) within the reference image (REF) and a second region of interest (ROI2_1) within the input image (INPUT).

[0093] First, referring to FIGS. 2 to 4, the image comparison unit (221) may include an ROI conversion unit (2211), a first template image generation unit (2212), a second template image generation unit (2213), a template image matching unit (2214), and a trigger signal generation unit (2215).

[0094] The ROI transformation unit (2211) can generate second region of interest information for a second region of interest (ROI2) in an input image (INPUT) based on first region of interest information for a first region of interest (ROI1) in a reference image (REF).

[0095] In one embodiment, the ROI transformation unit (2211) can transform the area of ​​the first region of interest (ROI1) to generate a second region of interest (ROI2). For example, the ROI transformation unit (2211) can expand the first region of interest (ROI1) to generate a second region of interest (ROI2).

[0096] To explain more specifically, with reference to FIG. 5, the first region of interest (ROI1) can be defined in the reference image (REF). For example, the first region of interest (ROI1) is used for the image comparison unit (221) to determine whether the input image (INPUT) and the reference image (REF) match, and the first region of interest (ROI1) can be set to overlap with at least some area within the reference image (REF).

[0097] The first region of interest (ROI1) may be provided in various shapes. For example, as illustrated in FIG. 5, the first region of interest (ROI1) may be provided in a rectangular or square shape having two pairs of sides parallel to each other, but the embodiments of the present disclosure are not limited thereto. According to the embodiment, the first region of interest (ROI1) provided in a rectangular or square shape may have a rounded corner where one long side and one short side meet. However, the embodiments of the present disclosure are not limited thereto, and the first region of interest (ROI1) may have various planar shapes such as a polygon or a circle.

[0098] The first region of interest information may include information (e.g., coordinate information) regarding a first reference point (PT1) within the first region of interest (ROI1) and information regarding a first area (A1) of the first region of interest (ROI1). For example, the first reference point (PT1) may be set as the center point of the first region of interest (ROI1) having a rectangular or square shape. However, embodiments of the present disclosure are not limited thereto, and the first reference point (PT1) may be described as various locations (e.g., vertices, etc.) within the first region of interest (ROI1). Here, since the first region of interest (ROI1) has a rectangular or square shape, the first region of interest (ROI1) may be extracted (or derived, generated) based on information regarding the first reference point (PT1) within the first region of interest (ROI1) and information regarding the first area (A1) of the first region of interest (ROI1).

[0099] However, this is merely illustrative, and the information included in the first region of interest information is not limited thereto. For example, the first region of interest information may include information about one vertex of the first region of interest (ROI1) having a rectangular or square shape and another vertex facing diagonally from said vertex. As another example, the first region of interest information may include information about one diagonal of the first region of interest (ROI1) having a rectangular or square shape and information about the angle formed by said diagonal with one side (e.g., a short side or a long side) of the first region of interest (ROI1). As yet another example, the first region of interest information may include information about one short side and one long side of the first region of interest (ROI1) having a rectangular or square shape. That is, the first region of interest information may include various information from which the first region of interest (ROI1) can be extracted (or derived).

[0100] Referring to FIG. 4, as described above, the ROI transformation unit (2211) can transform, for example, expand the area of ​​the first region of interest (ROI1) to create a second region of interest (ROI2). For example, the ROI transformation unit (2211) can transform (i.e., expand) the area of ​​the first region of interest (ROI1) to create a second region of interest (ROI2) such that the center point of the second region of interest (ROI2) (e.g., second reference point (PT2)) in image coordinates has the same coordinate value as the center point of the first region of interest (ROI1) (e.g., first reference point (PT1)), and the first region of interest (ROI1) is included in the second region of interest (ROI2). Accordingly, at least a portion of the area of ​​the second region of interest (ROI2) may overlap with the first region of interest (ROI1).

[0101] For example, as described below, the template image matching unit (2214) receives a first template image (TPIG1) (or a first image segment) in an area overlapping with a first region of interest (ROI1) in a reference image (REF) and a second template image (TPIG2) (or a second image segment) in an area overlapping with a second region of interest (ROI2) in an input image (INPUT), and can search for a matching position of the first template image (TPIG1) within the second template image (TPIG2). As an example, the template image matching unit (2214) can search for a matching position of the first template image (TPIG1) within the second template image (TPIG2) by utilizing a template matching technique. In this way, since the matching position must be searched while the first template image (TPIG1) is sliding within the second template image (TPIG2), the second template image (TPIG2) needs to be set larger than the first template image (TPIG1).

[0102] In addition, the second template image (TPIG2) needs to be set larger than the first template image (TPIG1) to reflect the change in the camera's viewpoint and / or the degree to which the camera may be tilted to acquire the input image (INPUT).

[0103] More specifically, in order to locate the matching position of the first template image (TPIG1) within the second template image (TPIG2), it is necessary for the second template image (TPIG2) to include an image (e.g., an image segment) that matches an image (e.g., an image segment) corresponding to at least a portion of the area within the first template image (TPIG1). That is, even if changes in camera viewpoint or camera distortion occur, the first region of interest (ROI1) within the reference image (REF) and the second region of interest (ROI2) within the input image (INPUT) need to overlap in at least a portion.

[0104] However, if the width of the second template image (TPIG2) is set to be equal to or smaller than the width of the first template image (TPIG1), for example, if the width of the second region of interest (ROI2) is set to be equal to or smaller than the width of the first region of interest (ROI1), if camera viewpoint changes or camera distortion occur relatively severely, there may not be a matching part (e.g., an overlapping image or image segment) between the second template image (TPIG2) and the first template image (TPIG1) depending on the case (e.g., there is no overlapping area between the second region of interest (ROI2) and the first region of interest (ROI1). In this case, the template image matching unit (2214) may not be able to search for the matching position of the first template image (TPIG1) within the second template image (TPIG2).

[0105] Accordingly, in an embodiment of the present disclosure, the ROI transformation unit (2211) can generate a second region of interest (ROI2) by expanding the area of ​​the first region of interest (ROI1) to a preset range. For example, as illustrated in FIGS. 5 and 6a, the ROI transformation unit (2211) can generate the second region of interest (ROI2a) such that it has a second reference point (PT2) having the same coordinates as the first reference point (PT1) of the first region of interest (ROI1), and the second area (A2) of the second region of interest (ROI2a) is set to be larger than the first area (A1) of the first region of interest (ROI1).

[0106] Here, the second area (A2) can be extracted (or calculated) by multiplying the first area (A1) by a preset extension value (wherein the extension value is a real number greater than 1) considering the location where the camera is installed, the environment of the product production line, etc. For example, the second area (A2) may be 1.5 times the first area (A1), but is not limited thereto.

[0107] Referring again to FIG. 4, the first template image generation unit (2212) receives first region of interest information regarding a reference image (REF) and a first region of interest (ROI1), and can generate a first template image (TPIG1) based on the reference image (REF) and the first region of interest information.

[0108] For example, referring together with FIG. 5, the first template image generation unit (2212) can generate a first template image (TPIG1) by extracting an image, for example, an image segment (or template image), that overlaps with the first region of interest (ROI1) within the reference image (REF). That is, the area where the first template image (TPIG1) is placed and the first region of interest (ROI1) can overlap. For example, the area where the first template image (TPIG1) is placed and the first region of interest (ROI1) may be substantially the same area. In other words, the first template image (TPIG1) and the first region of interest (ROI1) have the same shape (e.g., rectangular shape or square shape), the first center point (CT1) of the first template image (TPIG1) and the first reference point (PT1) of the first region of interest (ROI1) have the same coordinates within the reference image (REF), and the area of ​​the first template image (TPIG1) may be the same as the first area (A1) of the first region of interest (ROI1).

[0109] Additionally, referring to FIG. 4, the second template image generation unit (2213) receives second region of interest information regarding the input image (INPUT) and the second region of interest (ROI2), and can generate a second template image (TPIG2) based on the input image (INPUT) and the second region of interest information.

[0110] For example, referring together with FIG. 6a, the second template image generation unit (2213) can generate a second template image (TPIG2a) by extracting an image, for example, an image segment (or template image), that overlaps with the second region of interest (ROI2a) within the input image (INPUTa). That is, the area where the second template image (TPIG2a) is placed and the second region of interest (ROI2a) can overlap. For example, the area where the second template image (TPIG2a) is placed and the second region of interest (ROI2a) may be substantially the same area. In other words, the second template image (TPIG2a) and the second region of interest (ROI2a) have the same shape (e.g., rectangular shape or square shape), the second center point (CT2a) of the second template image (TPIG2a) and the second reference point (PT2) of the second region of interest (ROI2a) have the same coordinates within the input image (INPUTa), and the area of ​​the second template image (TPIG2a) may be the same as the second area (A2) of the second region of interest (ROI2a).

[0111] Referring again to FIG. 4, the first template image (TPIG1) generated by the first template image generation unit (2212) and the second template image (TPIG2) generated by the second template image generation unit (2213) can be provided to the template image matching unit (2214).

[0112] The template image matching unit (2214) can calculate (or generate, extract) a similarity score (MRD) based on the first template image (TPIG1) and the second template image (TPIG2). For example, the template image matching unit (2214) can extract or search for matching locations between the first template image (TPIG1) and the second template image (TPIG2) and calculate a similarity score based thereon.

[0113] In one embodiment, the template image matching unit (2214) can calculate a similarity score (MRD) between the first template image (TPIG1) and the second template image (TPIG2) (or a similarity score between the reference image (REF) and the input image (INPUT)) using a template matching technique. For example, the template image matching unit (2214) can calculate the similarity score (MRD) by searching for a matching location of the first template image (TPIG1) within the second template image (TPIG2).

[0114] For example, the template image matching unit (2214) can slide the first template image (TPIG1) on the second template image (TPIG2) at regular intervals and calculate the similarity or error between the first template image (TPIG1) and the second template image (TPIG2) at each position. As an example, the template image matching unit (2214) can calculate the similarity or error by sliding the first template image (TPIG1) on the second template image (TPIG2) at regular intervals and comparing the pixel values ​​of the first template image (TPIG1) and the second template image (TPIG2) at each position (e.g., comparing pixel values ​​at each coordinate within the area where the first template image (TPIG1) is placed). And, the template image matching unit (2214) can extract the location with the highest similarity (or the location with the lowest error) among all locations where similarity or error has been calculated, and calculate (or extract, generate) a similarity score (MRD) between the first template image (TPIG1) and the second template image (TPIG2) based on the distance between the coordinates of the center point (first center point) of the first template image (TPIG1) and the coordinates of the center point (second center point) of the second template image (TPIG2) at the corresponding location within the second template image (TPIG2). For example, the smaller the distance between the coordinates of the first center point of the first template image (TPIG1) and the coordinates of the second center point of the second template image (TPIG2), the higher the similarity score (MRD) can be calculated.

[0115] For example, referring to FIG. 5 and FIG. 6a, the template image matching unit (2214) slides the first template image (TPIG1) on the second template image (TPIG2a) at regular intervals and calculates the similarity or error between the first template image (TPIG1) and the second template image (TPIG2a) at each position, and calculates (or extracts, generates) the similarity score (MRD) between the first template image (TPIG1) and the second template image (TPIG2a) based on the distance between the coordinates of the first center point (CT1) of the first template image (TPIG1) and the coordinates of the second center point (CT2a) of the second template image (TPIG2a) at the position with the highest similarity (or the position with the lowest error) within the second template image (TPIG2a).

[0116] Here, as shown in FIGS. 5 and 6a, since the distance between the coordinates of the first center point (CT1) of the first template image (TPIG1) and the coordinates of the second center point (CT2a) of the second template image (TPIG2a) at the location where the first template image (TPIG1) is matched is relatively small, the similarity score (MRD) can have a relatively high value.

[0117] In addition, as another example, referring to FIG. 5 and FIG. 6b, the template image matching unit (2214) slides the first template image (TPIG1) on the second template image (TPIG2b) at regular intervals, calculates the similarity or error between the first template image (TPIG1) and the second template image (TPIG2b) at each position, and calculates (or extracts, generates) the similarity score (MRD) between the first template image (TPIG1) and the second template image (TPIG2b) based on the distance between the coordinates of the first center point (CT1) of the first template image (TPIG1) and the coordinates of the second center point (CT2b) of the second template image (TPIG2b) at the position with the highest similarity (or the position with the lowest error) within the second template image (TPIG2b).

[0118] Here, as illustrated in FIGS. 5 and 6b, since the distance between the coordinates of the first center point (CT1) of the first template image (TPIG1) and the coordinates of the second center point (CT2b) of the second template image (TPIG2b) at the location where the first template image (TPIG1) is matched is relatively large, the similarity score (MRD) may have a relatively low value.

[0119] In this way, the template image matching unit (2214) can calculate (or extract, generate) the similarity score (MRD) between the first template image (TPIG1) and the second template image (TPIG2) (or the similarity score between the reference image (REF) and the input image (INPUT)) by using a template matching technique that searches for the image (or image segment) with the highest similarity (e.g., matching) to the first template image (TPIG1) within the second template image (TPIG2).

[0120] However, the embodiments of the present disclosure are not limited thereto, and the template image matching unit (2214) may calculate a similarity score (MRD) in various ways. For example, at least one alignment mark is formed on a product production line, and the template image matching unit (2214) may calculate a similarity score (MRD) by comparing the coordinates of the position where the alignment mark is placed in the first template image (TPIG1) with the coordinates of the position where the alignment mark is placed in the second template image (TPIG2).

[0121] Meanwhile, in order for the template image matching unit (2214) to search for the matching position of the first template image (TPIG1) within the second template image (TPIG2), as shown in FIGS. 5 and 6a, the first region of interest (ROI1) and the second region of interest (ROI2) can each be set so as not to overlap with the object region (OBA) where an object image (OBJ) corresponding to a product existing on the product production line among the reference image (REF) or input image (INPUT) can be placed.

[0122] For example, when capturing images using a camera on a product production line, particularly a product production line where a high-speed process takes place, the position of the object image (OBJ) within the image may change in real time as the product to be inspected moves in real time. For example, within the reference image (REF) and / or input image (INPUT), the object image (OBJ) may be placed on any area of ​​the object region (OBA).

[0123] Here, if the first region of interest (ROI1) and / or the second region of interest (ROI2) overlap with the object region (OBA) in at least part, then, depending on the case, at least part of the first template image (TPIG1) generated based on the first region of interest (ROI1) and the second template image (TPIG2) generated based on the second region of interest (ROI2) may include an image segment corresponding to at least part of the object image (OBJ). In this case, as described above, since the object image (OBJ) is not an image segment existing at a fixed location but rather one whose position changes in real time depending on the time of shooting, the similarity calculation may be inaccurate when the template image matching unit (2214) searches for the matching position of the first template image (TPIG1) within the second template image (TPIG2).

[0124] Accordingly, in an embodiment of the present disclosure, the first region of interest (ROI1) and the second region of interest (ROI2) may each be configured to be non-overlapping with the object region (OBA) where the object image (OBJ) can be placed.

[0125] However, this is merely illustrative and the embodiments of the present disclosure are not limited thereto. For example, referring to FIG. 7, at least one of the first region of interest (ROI1_1) and the second region of interest (ROI2_1) may be set to overlap at least partially with the object region (OBA) where the object image (OBJ) can be placed. As an example, at least one of the first center point (PT1_1) of the first region of interest (ROI1_1) and the second center point (PT2_1) of the second region of interest (ROI2_1) may be placed within the object region (OBA).

[0126] Referring again to FIG. 4, the trigger signal generation unit (2215) can detect (or determine) whether an image error has occurred based on the similarity score (MRD) generated by the template image matching unit (2214).

[0127] In one embodiment, the trigger signal generation unit (2215) can detect whether an image error has occurred by comparing a similarity score (MRD) with a preset threshold value. For example, if the similarity score (MRD) is less than the threshold value, the trigger signal generation unit (2215) may determine that an image error has occurred due to a change in the camera's viewpoint or camera tilt (or determine that the image error has exceeded a preset range), and if the similarity score (MRD) is greater than or equal to the threshold value, it may determine that no image error has occurred (or determine that the image error is within a preset range). As an example, the aforementioned preset threshold value may be determined experimentally after performing a product appearance inspection using images taken at various locations, but is not limited thereto, and may also be determined by a separate mathematical formula and / or function.

[0128] Additionally, the trigger signal generation unit (2215) may generate and / or output a trigger signal (TRG) to activate the image correction unit (222) when it is determined that an image error has occurred, for example, when the similarity score (MRD) is below a threshold value. On the other hand, the trigger signal generation unit (2215) may not generate (or output) a trigger signal (TRG) when it is determined that no image error has occurred, for example, when the similarity score (MRD) is above a threshold value.

[0129] For example, as shown in FIGS. 5 and 6a, when a similarity score (MRD) having a relatively high value is calculated according to the alignment position of the first template image (TPIG1) and the second template image (TPIG2a), for example, when the calculated similarity score (MRD) is greater than or equal to a threshold value, the trigger signal generation unit (2215) may not generate (or output) a trigger signal (TRG).

[0130] In contrast, as shown in FIGS. 5 and 6b, when a similarity score (MRD) having a relatively low value is calculated according to the alignment position of the first template image (TPIG1) and the second template image (TPIG2b), for example, when the calculated similarity score (MRD) is below a threshold value, the trigger signal generation unit (2215) can generate (or output) a trigger signal (TRG).

[0131] However, it is not limited thereto, and according to the embodiment, the trigger signal generation unit (2215) may generate (or output) a trigger signal (TRG) that instructs activation (e.g., a trigger signal (TRG) having a value of 1) when it is determined that an image error has occurred, for example, when the similarity score (MRD) is below a threshold value, and may generate (or output) a trigger signal (TRG) that instructs deactivation (e.g., a trigger signal (TRG) having a value of 0) when it is determined that no image error has occurred, for example, when the similarity score (MRD) is above a threshold value.

[0132] For example, as shown in FIGS. 5 and 6a, when a similarity score (MRD) having a relatively high value is calculated according to the alignment position of the first template image (TPIG1) and the second template image (TPIG2a), for example, when the calculated similarity score (MRD) is greater than or equal to a threshold value, the trigger signal generator (2215) can generate (or output) a trigger signal (TRG) that indicates deactivation (e.g., a trigger signal (TRG) having a value of 0).

[0133] In contrast, as shown in FIGS. 5 and 6b, when a similarity score (MRD) having a relatively low value is calculated according to the alignment position of the first template image (TPIG1) and the second template image (TPIG2b), for example, when the calculated similarity score (MRD) is below a threshold value, the trigger signal generator (2215) can generate (or output) a trigger signal (TRG) that indicates activation (e.g., a trigger signal (TRG) having a value of 1).

[0134] In this way, the image correction unit (222), which is a subsequent module of the image comparison unit (221), can be activated or deactivated according to the instruction value of the trigger signal (TPG).

[0135] As described with reference to FIGS. 4 to 7, the image comparison unit (221) generates a trigger signal (TRG) based on an input image (INPUT) and a reference image (REF), and depending on whether the trigger signal (TPG) is generated and / or output, the image correction unit (222), which is a subsequent module of the image comparison unit (221), may be activated or deactivated. That is, according to the embodiments of the present disclosure, the image correction unit (222) is activated only when an image error occurs between the input image (INPUT) and the reference image (REF) and correction of the input image (INPUT) is required, so the system processing speed may be reduced and unnecessary resource consumption may be prevented, and accordingly, system operation efficiency may be improved.

[0137] <영상 보정부의 실시예>

[0138] FIG. 8 is a block diagram showing an example of an image correction unit (222) included in the corrector (220) of FIG. 3a.

[0139] Referring to FIGS. 2, FIGS. 3a and FIG. 8, the image correction unit (222) can be activated or deactivated according to a trigger signal (TRG). For example, when the image correction unit (222) is activated, the image correction unit (222) can correct the input image (INPUT) to generate and / or output a corrected image (OUT). On the other hand, when the image correction unit (222) is deactivated, the image correction unit (222) can output the input image (INPUT) as a corrected image (OUT) without any separate operation (driving) (e.g., bypass), or the input image (INPUT) input to the image correction system (200), e.g., the corrector (220), can be output as is without passing through the image comparison unit (221).

[0140] To explain the operation of the image correction unit (222) more specifically when the image correction unit (222) is activated, refer to FIG. 8. The image correction unit (222) can generate a corrected image (OUT) by correcting the input image (INPUT) so that it is matched (or aligned) with the reference image (REF). For example, the image correction unit (222) can quantitatively analyze the difference in viewpoint between the reference image (REF) and the input image (INPUT) and correct the input image (INPUT) so that it is matched (or aligned) with the reference image (REF) through geometric transformation. That is, the image correction unit (222) can generate a corrected image (OUT) by correcting (or transforming) the input image (INPUT) so that it can be treated as having been taken at the same viewpoint as the reference image (REF).

[0141] To this end, in one embodiment, the image correction unit (222) may include a first feature point extraction unit (2221), a second feature point extraction unit (2222), and an output unit (2223). In one embodiment, the image correction unit (222) may further include a storage unit (2224).

[0142] The first feature point extraction unit (2221) can extract a first feature point (FT1) based on a reference image (REF). For example, the first feature point extraction unit (2221) can extract a point that is a feature within the reference image (REF), which can be treated as an image feature point, for example, as the first feature point (FT1). For example, the first feature point extraction unit (2221) can extract a point corresponding to a region(s) where a brightness difference occurs, a color difference occurs, a part where the image changes abruptly, a vertex, an edge, a corner, a boundary, etc., as the first feature point (FT1), but is not limited thereto.

[0143] The second feature point extraction unit (2222) can extract a second feature point (FT2) based on the input image (INPUT). For example, the second feature point extraction unit (2222) can extract a point that is a feature within the input image (INPUT), which can be treated as an image feature point within the input image (INPUT), as the second feature point (FT2). For example, the second feature point extraction unit (2222) can extract a point corresponding to a region(s) where a brightness difference occurs, a color difference occurs, a part where the image changes abruptly, a vertex, an edge, a corner, a boundary, etc., within the input image (INPUT) as the second feature point (FT2), but is not limited thereto.

[0144] According to the embodiment, the first feature point extraction unit (2221) and the second feature point extraction unit (2222) can extract feature points using various known feature point extraction algorithms and / or artificial intelligence models.

[0145] Meanwhile, in FIG. 8, the first feature point extraction unit (2221) and the second feature point extraction unit (2222) are shown as separate components, but this is merely an example for convenience of explanation, and according to the embodiment, the first feature point extraction unit (2221) and the second feature point extraction unit (2222) can be integrated and implemented as a single feature point extraction unit.

[0146] Additionally, the output unit (2223) receives the first feature point (FT1) and the second feature point (FT2) from the first feature point extraction unit (2221) and the second feature point extraction unit (2222), and can correct (or convert) the input image (INPUT) based on the first feature point (FT1) and the second feature point (FT2) to generate a corrected image (OUT).

[0147] In one embodiment, the output unit (2223) can form a matching pair by matching a first feature point (FT1) and a second feature point (FT2). For example, the output unit (2223) can extract a first descriptor for the first feature point (FT1) and a second descriptor for the second feature point (FT2), and form a matching pair using the first descriptor and the second descriptor.

[0148] Here, the first descriptor and the second descriptor may refer to feature vectors related to an image as image descriptors. For example, the first descriptor and the second descriptor may be generated by assigning directional information to the first feature point and the second feature point, respectively, through differentiation. Meanwhile, in this specification, a descriptor may be referred to as a 'feature point to which directional information is assigned'.

[0149] However, this is merely illustrative and the embodiments of the present disclosure are not limited thereto. For example, the output unit (2223) may generate a first descriptor and a second descriptor by vectorizing local patterns from each of the first feature point (FT1) and the second feature point (FT2). For example, the output unit (2223) may generate the first descriptor and the second descriptor by generating a feature vector by calculating the relationship with surrounding pixels for each of the first feature point (FT1) and the second feature point (FT2).

[0150] The output unit (2223) can form a matching pair of the first feature point (FT1) and the second feature point (FT2) by comparing and / or matching the first descriptor for the first feature point (FT1) and the second descriptor for the second feature point (FT2).

[0151] In this way, the output unit (2223) matches the first feature point (FT1) and the second feature point (FT2) using the first descriptor for the first feature point (FT1) and the second descriptor for the second feature point (FT2), that is, the first descriptor and the second descriptor to which directional information is assigned, and forms a matching pair of the first feature point (FT1) and the second feature point (FT2). Therefore, even when the input image (INPUT) acquired by the camera rotates, tilts, or becomes distorted due to changes in the camera's viewpoint or tilting, all directional information is maintained, so robust characteristics against image rotation or tilting can be secured, and image matching can be performed more easily and accurately.

[0152] However, this is merely illustrative and the embodiments of the present disclosure are not limited thereto, and the output unit (2223) may form a matching pair with the first feature point (FT1) and the second feature point (FT2) themselves.

[0153] According to an embodiment, the first feature point extraction unit (2221) and the second feature point extraction unit (2222) extract the first feature point (FT1) and the second feature point (FT2), and the output unit (2223) matches the first feature point (FT1) and the second feature point (FT2) (e.g., matching the first descriptor and the second descriptor) to form matching pairs, the image correction unit (222) may use various techniques, algorithms, and / or artificial intelligence models. For example, for the aforementioned feature point extraction and feature point matching, the image correction unit (222) may use mathematical calculation methods such as ORB (oriented FAST and rotated BRIEF), SIFT (scale-invariant feature transform), SURF (speeded up robust features), etc., and / or deep learning models such as LIFT (learned invariant feature transform) and SuperPoint (self-supervised interest point detection and description), but is not limited thereto.

[0154] Additionally, the output unit (2223) can generate (or calculate, compute, extract) a transformation matrix (HMM) based on matching pairs of the first feature point (FT1) and the second feature point (FT2). For example, the output unit (2223) can generate a transformation matrix (HMM) based on matching pairs of the first feature point (FT1) and the second feature point (FT2) to transform the second feature point (FT2) of the input image (INPUT) so that it corresponds to the first feature point (FT1) of the reference image (REF). As an example, the output unit (2223) can generate a transformation matrix (HMM) such that the first feature point (FT1) can be derived when the transformation matrix (HMM) is applied (e.g., matrix multiplication) to the second feature point (FT2).

[0155] Meanwhile, the transformation matrix (HMM) is a matrix used in image registration techniques and can be defined as a 3x3 matrix representing a perspective transform on a two-dimensional (2D) plane, which can include rotation, translation, scaling, and perspective transformation (projection). For example, the transformation matrix (HMM) may be a homography matrix, but is not limited thereto.

[0156] Meanwhile, the output unit (2223) may generate a transformation matrix (HMM) by removing outliers using a separate algorithm and / or artificial intelligence model (e.g., RANSAC (Random Sample Consensus) algorithm, etc.) when generating the transformation matrix (HMM), but is not limited thereto.

[0157] Additionally, the output unit (2223) can generate a corrected image (OUT) by correcting (or transforming) the input image (INPUT) using the generated transformation matrix (HMM). For example, the output unit (2223) can generate a corrected image (OUT) by correcting the input image (INPUT) to match (or align) the reference image (REF) by performing a geometric transformation (warp transform) on the transformation matrix (HMM).

[0158] According to an embodiment, the output unit (2223) may generate a corrected image (OUT) by applying a transformation matrix (HMM) to the entire input image (INPUT), but is not limited thereto. For example, the output unit (2223) may generate a corrected image (OUT) by applying a transformation matrix (HMM) only to images (or image segments) that correspond (overlap) to the region of interest (or inspection region of interest) required for product appearance inspection among the input image (INPUT). As an example, the inspection region of interest may be set as an object region (OBA) where an object image (OBJ) corresponding to a product existing on the aforementioned product production line can be placed, but is not limited thereto. In this way, when the output unit (2223) generates a corrected image (OUT) by applying a transformation matrix (HMM) only to images corresponding to a part of the input image (INPUT) (i.e., the inspection region of interest), the operation of the output unit (2223) is simplified and system efficiency can be further improved.

[0159] Additionally, according to an embodiment, the transformation matrix (HMM) generated from the output unit (2223) may be stored inside the image correction unit (222), for example, in the storage unit (2224). Subsequently, when the input image (INPUT) is similar to a reference, the image correction unit (222) can correct the input image (INPUT) using the transformation matrix (HMM) stored in the storage unit (2224) without recalculation. Accordingly, the operation of the image correction unit (222) can be simplified, and system efficiency can be improved.

[0161] <본 개시의 실시예들에 따른 방법>

[0162] FIG. 9 is a flowchart illustrating an image correction method according to various embodiments of the present disclosure.

[0163] For example, FIG. 9 shows a flowchart illustrating an inference operation (correction operation) of an image correction system (200) (e.g., a corrector (220)) according to embodiments of the present disclosure.

[0164] Referring to FIGS. 1 through 9, the image correction method of FIG. 9 can be performed on the image correction system (200) (or electronic device (100)) described with reference to FIGS. 1 through 8. For example, the image correction method of FIG. 9 may be substantially identical to the inference operation (correction operation) of the image correction system (200) (or electronic device (100)) described with reference to FIGS. 1 through 8, that is, the operation of the corrector (220). Accordingly, redundant descriptions will not be repeated.

[0165] Referring to FIG. 9, the image correction method of FIG. 9 may include the step of generating a first template image based on a reference image and first region of interest information (S910), the step of generating a second template image based on an input image and second region of interest information (S920), and the step of generating a trigger signal based on the first template image and the second template image (S930). For example, the step of generating a first template image (S910) based on a reference image and first region of interest information is substantially the same or similar to the operation of the first template image generating unit (2212) of the image comparison unit (221) described with reference to FIGS. 2 to 8, the step of generating a second template image (S920) based on an input image and second region of interest information is substantially the same or similar to the operation of the ROI conversion unit (2211) and the second template image generating unit (2213) of the image comparison unit (221) described with reference to FIGS. 2 to 8, and the step of generating a trigger signal (S930) based on the first template image and the second template image may be substantially the same or similar to the operation of the template image matching unit (2214) and the trigger signal generating unit (2215) of the image comparison unit (221) described with reference to FIGS. 2 to 8.

[0166] Additionally, the image correction method of FIG. 9 may further include a step of outputting a corrected image. For example, in the step of outputting the corrected image, the image correction method of FIG. 9 may determine whether a trigger has occurred according to a trigger signal (S940), and if a trigger has occurred (indicated as "Yes" in FIG. 9), compare a reference image with an input image, correct the input image based on the comparison result to generate a corrected image, and if a trigger has not occurred (indicated as "No" in FIG. 9), output the input image as a corrected image (e.g., bypass).

[0168] <본 개시의 실시예들에 따른 컴퓨터 프로그램>

[0169] The embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions may include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa.

[0170] Meanwhile, at least some of the methods described above (e.g., the method of FIG. 9) may be implemented by hardware, software, firmware, middleware, microcode, a hardware description language, or a combination of two or more of the above. Where at least some of the methods described above are implemented by software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks may be stored on a machine or computer-readable medium, such as a storage medium, thereby enabling a processor to perform the necessary tasks.

[0172] <본 개시의 실시예들에 따른 다양한 효과>

[0173] As described above, the image correction system, device, server, method, and computer program according to the embodiments of the present disclosure can detect whether an image error, such as a change in camera viewpoint or camera distortion, has occurred by comparing an input image with a reference image.

[0174] In addition, the image correction system, device, server, method, and computer program according to the embodiments of the present disclosure correct the input image to match (or align) the reference image using geometric-based alignment when an image error occurs, thereby enabling more precise correction of the input image. Accordingly, since the input image used for product appearance inspection can be corrected to correspond to the viewpoint of the reference image regardless of changes or distortions in the camera viewpoint, the reliability and accuracy of the inspection can be improved during product appearance inspection using the image.

[0175] Herein, an image correction system, device, server, method, and computer program according to embodiments of the present disclosure correct an input image to match (or align) a reference image only when an image error occurs (or when the image error exceeds a preset range), and can deactivate a module for image correction (e.g., an image correction unit) when no image error occurs (or when the image error is within a preset range). That is, since a subsequent module for image correction (e.g., an image correction unit) is activated and operates only when correction is required, a decrease in system processing speed and unnecessary resource consumption can be prevented, and accordingly, system operational efficiency can be improved.

[0176] In addition, when using an image correction system, device, server, method, and computer program according to the embodiments of the present disclosure, hardware equipment required in the prior art is not separately required, so adaptability to changes in the image shooting environment is ensured, robustness to environmental changes is possible, and dependence on external equipment for correcting input images can be minimized.

[0177] However, the effects of the present disclosure are not limited to those exemplified above, and various other effects are included within the present disclosure and may be extended in various ways without departing from the spirit and scope of the present disclosure.

[0179] <본 개시의 다양한 실시예>

[0180] A system according to various embodiments of the present disclosure can be described as follows.

[0181] In one embodiment of the present disclosure, the system comprises, in a system for correcting an input image, an image comparison unit that generates a first template image based on first region of interest information including information about a reference image and a first region of interest, generates a second template image based on the input image, and generates a trigger signal based on the first template image and the second template image, and an image correction unit that compares the reference image and the input image and corrects the input image based on the comparison result to generate a corrected image, wherein the image correction unit may be activated or deactivated according to the trigger signal.

[0182] In one embodiment, the image comparison unit may include an ROI transformation unit that generates a second region of interest by transforming the first region of interest based on the first region of interest information; a first template image generation unit that generates a first template image by extracting an image that overlaps with the first region of interest among the reference image based on the reference image and the first region of interest information; a second template image generation unit that generates a second template image by extracting an image that overlaps with the second region of interest among the input image based on the second region of interest information including the input image and information about the second region of interest; a template image matching unit that calculates a similarity score by extracting a matching position of the first template image on the second template image; and a trigger signal generation unit that generates the trigger signal by comparing the similarity score with a preset threshold value.

[0183] In one embodiment, the second area of ​​the second area of ​​interest may be larger than the first area of ​​the first area of ​​interest.

[0184] In one embodiment, when the similarity score is greater than or equal to the threshold value, the image correction unit is deactivated according to the trigger signal, and when the similarity score is less than the threshold value, the image correction unit may be activated according to the trigger signal.

[0185] In one embodiment, the image correction unit may include a first feature point extraction unit that extracts a first feature point based on the reference image, a second feature point extraction unit that extracts a second feature point based on the input image, and an output unit that matches the first feature point and the second feature point to form a match pair, generates a transformation matrix using the match pair, and corrects the input image using the transformation matrix to generate the corrected image.

[0186] In one embodiment, the output unit may generate the corrected image by applying the transformation matrix to the entire input image.

[0187] In one embodiment, the output unit may generate the correction image by applying the transformation matrix to an image corresponding to an inspection interest region among the input images.

[0188] In one embodiment, when the image correction unit is deactivated, the input image may be output as the correction image.

[0189] An apparatus according to various embodiments of the present disclosure can be described as follows.

[0190] In one embodiment of the present disclosure, the device comprises, in a device for correcting an input image, a memory for storing at least one instruction, and a processor for performing at least one process for correcting the input image in response to the instruction stored on the memory, wherein the processor may generate a first template image based on first region of interest information including information about a reference image and a first region of interest, generate a second template image based on the input image, and generate a trigger signal based on the first template image and the second template image. Depending on the trigger signal, the processor may i) compare the reference image and the input image and correct the input image based on the comparison result to generate and output a corrected image, or ii) output the input image as the corrected image.

[0191] Computer programs according to various embodiments of the present disclosure may be described as follows.

[0192] In one embodiment of the present disclosure, a computer program is a computer program stored on a computer-readable medium, and when executed by a computer system, the computer system may include instructions for performing a method comprising: generating a first template image based on first region of interest information including information about a reference image and a first region of interest; generating a second template image based on an input image; generating a trigger signal based on the first template image and the second template image; and outputting a correction image. The step of outputting the correction image may, depending on the trigger signal, i) compare the reference image and the input image and correct the input image based on the comparison result to generate and output the correction image, or ii) output the input image as the correction image.

[0193] Although the present disclosure has been described with reference to embodiments thereof, those skilled in the art will understand that various modifications and changes can be made to the present disclosure without departing from the spirit and scope of the disclosure as set forth in the claims. Explanation of the symbols

[0194] 100: Electronic device 110: Processor 120: Communication module 130: Input Module 140: Display Module 150: Memory 151: Program 200: Image Correction System 210: Learning device 220, 220_1: Corrector 221: Video Comparison Section 222: Image Correction Unit 223: Memory 2211: ROI conversion unit 2212: 1st Template Image Generation Unit 2213: 2nd Template Image Generation Unit 2214: Template Image Matching Section 2215: Trigger signal generation unit 2221: 1st feature point extraction unit 2222: Second feature point extraction unit 2223: Output section 2224: Storage section

Claims

Claim 1 An image comparison unit for a system for correcting an input image, which generates a first template image based on first region of interest information including information about a reference image and a first region of interest, generates a second template image based on the input image, and generates a trigger signal based on the first template image and the second template image; The system includes an image correction unit that compares the reference image and the input image and corrects the input image based on the comparison result to generate a corrected image, wherein the image correction unit is activated or deactivated according to the trigger signal, and the image comparison unit includes: an ROI transformation unit that transforms the first region of interest based on the first region of interest information to generate a second region of interest; a first template image generation unit that generates the first template image by extracting an image overlapping the first region of interest among the reference image based on the reference image and the first region of interest information; a second template image generation unit that generates the second template image by extracting an image overlapping the second region of interest among the input image based on the second region of interest information including the input image and information about the second region of interest; a template image matching unit that extracts a matching position of the first template image on the second template image and calculates a similarity score; and a trigger signal generation unit that generates the trigger signal by comparing the similarity score with a preset threshold value, wherein the second area of ​​the second region of interest is larger than the first area of ​​the first region of interest. Claim 2 delete Claim 3 delete Claim 4 A system according to claim 1, wherein if the similarity score is greater than or equal to the threshold value, the image correction unit is deactivated according to the trigger signal, and if the similarity score is less than the threshold value, the image correction unit is activated according to the trigger signal. Claim 5 A system according to claim 1, wherein the image correction unit comprises: a first feature point extraction unit that extracts a first feature point based on the reference image; a second feature point extraction unit that extracts a second feature point based on the input image; and an output unit that matches the first feature point and the second feature point to form a match pair, generates a transformation matrix using the match pair, and corrects the input image using the transformation matrix to generate the corrected image. Claim 6 In claim 5, the output unit generates the corrected image by applying the transformation matrix to the entire input image. Claim 7 In claim 5, the output unit generates the correction image by applying the transformation matrix to an image corresponding to an inspection interest region among the input images. Claim 8 A system according to claim 1, wherein when the image correction unit is deactivated, the input image is output as the correction image. Claim 9 In a device for correcting an input image, a memory for storing at least one instruction; The system includes a processor that performs at least one process for correcting the input image in response to the instruction stored in the memory, wherein the processor generates a first template image based on first region of interest information including information about a reference image and a first region of interest, generates a second template image based on the input image, and generates a trigger signal based on the first template image and the second template image, wherein the processor, in accordance with the trigger signal, i) compares the reference image and the input image and corrects the input image based on the comparison result to generate and output a corrected image, or ii) outputs the input image as the corrected image, wherein the processor generates a second region of interest by transforming the first region of interest based on the first region of interest information, generates the first template image by extracting an image that overlaps with the first region of interest among the reference image based on the reference image and the first region of interest information, and generates the second template image by extracting an image that overlaps with the second region of interest among the input image based on second region of interest information including information about the input image and the second region of interest, and the second template A device that extracts a matching location of the first template image on an image to calculate a similarity score, generates a trigger signal by comparing the similarity score with a preset threshold, and wherein the second area of ​​the second region of interest is larger than the first area of ​​the first region of interest. Claim 10 A computer program stored on a computer-readable medium, wherein when executed by a computer system, the computer system comprises the steps of: generating a first template image based on first region of interest information including information about a reference image and a first region of interest; generating a second template image based on an input image; and generating a trigger signal based on the first template image and the second template image. The method includes a command to perform a method including the step of outputting a corrected image, wherein, in the step of outputting the corrected image, according to the trigger signal, i) compare the reference image and the input image and correct the input image based on the comparison result to generate and output the corrected image, or ii) output the input image as the corrected image, and wherein, in the step of generating the first template image, the step of generating the second template image, and the step of generating the trigger signal, the method generates the second region of interest by transforming the first region of interest based on the first region of interest information, generates the first template image by extracting an image overlapping the first region of interest from the reference image based on the reference image and the first region of interest information, generates the second template image by extracting an image overlapping the second region of interest from the input image based on the second region of interest information including the input image and information about the second region of interest, calculates a similarity score by extracting the alignment location of the first template image on the second template image, generates the trigger signal by comparing the similarity score with a preset threshold value, and the second of the second region of interest A computer program whose area is larger than the first area of ​​the first region of interest.

Citation Information

Patent Citations

  • Substrate measuring device and laser processing system

    KR1020190017021A

  • A method for adjusting an inspection area and a device for adjusting an inspection area

    KR102477088B1