Image processing apparatus, image processing system, and image processing method

Abstract

The invention provides an image processing apparatus, an image processing system, and an image processing method. The image processing apparatus may include: an exclusion region determination module configured to distinguish a vehicle body region and a component region in a design data-based image, and determine a partial region including the component region as an exclusion region; a reference image generation module configured to generate a reference image, the reference image including a masking region, the masking region masking the excluded region based on the design data image; a feature point search area defining module configured to define a feature point search area in the acquired image by using the reference image; and a feature point detection module configured to detect a plurality of feature points in the feature point search area.

Description

Technical Field

[0001] The present disclosure relates to an image processing device, an image processing system and an image processing method. Background Art

[0002] Image-based detection methods are used to check whether vehicle parts are assembled and the assembly status of these vehicle parts during the vehicle manufacturing process. For example, it is possible to determine whether the vehicle parts have been assembled by using images taken before and after the vehicle parts are assembled. However, due to different lighting environments or errors in the acquisition equipment, errors may occur between the acquired images. In order to compensate for this, it is necessary to perform image alignment (or image matching) between the acquired images. If image matching is not performed correctly between the acquired images, when a part is identified and extracted from the acquired image as a part or a part assembly part, the image quality of the part may deteriorate. In order to perform image alignment in this way, feature point detection can be used. Summary of the invention

[0003] Technical issues

[0004] The present disclosure attempts to provide an image processing device, an image processing system, and an image processing method capable of improving the accuracy of image matching by improving the performance of feature point detection used in image matching.

[0005] Technical Solution

[0006] An image processing device may include: an exclusion area determination module, configured to distinguish between a vehicle body area and a component area in a design data image, and determine a partial area including the component area as an exclusion area; a reference image generation module, configured to generate a reference image, the reference image including a masking area, the masking area masks the exclusion area in the design data image; a feature point search area definition module, configured to define a feature point search area in a captured image by using the reference image; and a feature point detection module, configured to detect multiple feature points in the feature point search area.

[0007] The image processing device may further include: a design data-based image generation module, configured to generate a design data-based image by matching shooting coordinates with design data coordinates, the shooting coordinates corresponding to a shot image provided from a camera, the camera is configured to shoot the vehicle, and the design data coordinates correspond to the design data.

[0008] The exclusion region determination module may be configured to distinguish between the vehicle body region and the component region according to a value assigned to each vehicle body or component identification number in the design data.

[0009] The exclusion area may further include a background area, which is distinguished from the vehicle body area and the component area in the image based on the design data.

[0010] The exclusion area may further include a user-specified area which is an area specified by a user and which has a large change according to the environment in the image based on the design data.

[0011] The image processing device may further include: an image matching module configured to perform image matching on the collected image based on a plurality of feature points.

[0012] The image processing device may further include: a transformation matrix generation module, configured to generate a transformation matrix according to a position deviation from a previously acquired captured image when performing image matching; and an error analysis module, configured to analyze whether component values ​​of the transformation matrix correspond to a predefined error range.

[0013] The multiple feature points may include a first feature point and a second feature point different from the first feature point, and the image matching module may be configured to perform image matching on the captured image by using the second feature point when a component value of a transformation matrix generated based on image matching using the first feature point exceeds an error range.

[0014] The feature point search area definition module can be configured to change the feature point search area by increasing or decreasing at least a portion of the boundary line of the feature point search area in units of pixels when the component value of the transformation matrix exceeds a predefined error range, the feature point detection module can be configured to detect new feature points in the changed feature point search area, and the image matching module can be configured to perform image matching on the captured image based on the new feature points.

[0015] An image processing system may include: a mobile robot configured to photograph a vehicle at photographing coordinates determined in a pre-provided teaching; a first image processing device receiving a first photographed image from the mobile robot and configured to generate a reference image for feature point search by utilizing a design data-based image matched with the first photographed image; and a second image processing device receiving a second photographed image from the mobile robot and configured to perform image matching on the second photographed image based on a feature point search area defined in the reference image.

[0016] The first image processing device may include: an exclusion area determination module, configured to distinguish between a vehicle body area and a component area in a design data image, and determine a partial area including the component area as an exclusion area; and a reference image generation module, configured to generate a reference image, the reference image including a masked area, the masked area masks the exclusion area in the design data image.

[0017] The exclusion region determination module may be configured to distinguish between the vehicle body region and the component region according to a value assigned to each vehicle body or component identification number in the design data.

[0018] The second image processing apparatus may include: a feature point search area definition module configured to define a feature point search area in the second photographed image by using a reference image; and a feature point detection module configured to detect a plurality of feature points in the feature point search area.

[0019] The second image processing device may further include: an image matching module, configured to perform image matching on the second captured image based on multiple feature points; a transformation matrix generation module, configured to generate a transformation matrix according to a position deviation from a previously acquired second captured image when performing image matching; and an error analysis module, configured to analyze whether component values ​​of the transformation matrix correspond to a predefined error range.

[0020] The multiple feature points may include a first feature point and a second feature point different from the first feature point, and the image matching module may be configured to perform image matching on the second captured image by using the second feature point when a component value of a transformation matrix generated based on image matching using the first feature point exceeds an error range.

[0021] The feature point search area definition module can be configured to change the feature point search area by increasing or decreasing at least a portion of a boundary line of the feature point search area in units of pixels when a component value of the transformation matrix exceeds a predefined error range, the feature point detection module can be configured to detect new feature points in the changed feature point search area, and the image matching module can be configured to perform image matching on the second captured image based on the new feature points.

[0022] An image processing method may include: distinguishing a vehicle body area and a component area in a design data-based image; determining a partial area including the component area as an exclusion area; generating a reference image, the reference image including a masking area, the masking area masks the exclusion area in the design data-based image; defining a feature point search area in a captured image by using the reference image; and detecting a plurality of feature points in the feature point search area.

[0023] The image processing method may further include: performing image matching on the captured image based on multiple feature points; when performing image matching, generating a transformation matrix based on a position deviation from a previously captured captured image; and analyzing whether component values ​​of the transformation matrix correspond to a predefined error range.

[0024] The multiple feature points may include a first feature point and a second feature point different from the first feature point, and the image processing method may further include: when a component value of a transformation matrix generated based on image matching using the first feature point exceeds an error range, performing image matching on the captured image by using the second feature point.

[0025] The image processing method may further include: when a component value of the transformation matrix exceeds a predefined error range, changing the feature point search area by increasing or decreasing at least a portion of a boundary line of the feature point search area in units of pixels; detecting new feature points in the changed feature point search area; and performing image matching on the captured image based on the new feature points.

[0026] Beneficial Effects

[0027] According to an embodiment, by distinguishing between areas in an image where changes are relatively small according to the shooting environment (e.g., vehicle body areas) and areas where changes are relatively large (e.g., component assembly areas, areas sensitive to lighting changes, etc.), and extracting feature points from areas where changes are relatively small, the performance of feature point detection can be improved and the accuracy of image matching can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a diagram for explaining an image processing system according to an embodiment.

[0029] Figure 2 FIG. 1 is a diagram for explaining an image based on design data that can be used in the image processing system according to the embodiment.

[0030] Figures 3 to 5 FIG. 1 is a diagram for explaining determination of an exclusion area in the image processing system according to the embodiment.

[0031] Figure 6 is a diagram for explaining a reference image that can be used in the image processing system according to the embodiment.

[0032] Figure 7 and Figure 8 : is a diagram for explaining a feature point search area defined in a captured image by using a reference image in the image processing system according to the embodiment.

[0033] Fig. 9 It is a diagram for explaining the image processing method according to the embodiment.

[0034] Fig.10 is a diagram for explaining an image processing system according to an embodiment.

[0035] Fig.11 and Fig.12 A diagram for explaining the error range of image acquisition due to errors in the acquisition equipment.

[0036] Fig.13 and Fig.14 It is a diagram for explaining feature point detection when the error range is exceeded.

[0037] Fig.15 It is a diagram for explaining the image processing method according to the embodiment.

[0038] Fig.16 It is a diagram for explaining feature point detection when the error range is exceeded.

[0039] Fig.17 It is a diagram for explaining the image processing method according to the embodiment.

[0040] Fig.18 is a diagram for explaining an embodiment of a computing device for realizing an image processing apparatus, an image processing system, and an image processing method according to the embodiment. DETAILED DESCRIPTION

[0041] The present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the present disclosure are shown. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways, all without departing from the spirit or scope of the present disclosure. Therefore, the drawings and descriptions will be regarded as illustrative and non-restrictive in nature. Throughout the specification, the same reference numerals represent the same elements.

[0042] Throughout the specification and claims, when a component "includes" a certain element, unless otherwise specified, it means that other elements may be further included, rather than excluding other elements. Terms including ordinal numbers such as first, second, etc. will only be used to describe various constituent elements and will not be interpreted as limiting these constituent elements. These terms are only used to distinguish one constituent element from other constituent elements.

[0043] The terms "unit", "component" or "portion", "-device" and "module" in this specification refer to a unit that processes at least one function or operation, which can be implemented by hardware, software, or a combination of hardware and software. In addition, at least part of the configuration or function of the image processing device, image processing system, and image processing method according to the embodiments described below can be implemented as a program or software, and the program or software can be stored in a computer-readable medium.

[0044] Figure 1 is a diagram for explaining an image processing system according to an embodiment.

[0045] Reference Figure 1, the image processing system 1 according to the embodiment may include a first image processing apparatus 10 , a second image processing apparatus 20 , and a mobile robot 30 .

[0046] The mobile robot 30 can be a robot that can understand the environment of an industrial site, move around, and assist or perform tasks on behalf of people. Unlike an automated guided vehicle (AGV) that relies on a predefined path, the mobile robot 30 is equipped with various types of sensors including cameras, computer vision sensors, voice recognition sensors, and lidars, and uses computing technologies such as artificial intelligence and machine learning to collect data about the environment and make decisions on its own, such as setting a path. For example, the mobile robot 30 can be used in various production processes in a robot-automated manufacturing factory for various industrial products such as vehicles. In the following embodiments, the mobile robot 30 can shoot a vehicle at a shooting coordinate determined in a pre-provided teaching.

[0047] The first image processing device 10 can generate a reference image IMG2 by using a captured image IMG1 and a design data-based image. Here, the captured image IMG1 is an image acquired by pre-shooting at least a portion of the vehicle, and the design data-based image is an image generated based on design data (e.g., 3D cad data). Taking a door frame as an example, the captured image IMG1 is, for example, an image of a door frame of a vehicle captured by the mobile robot 30 in a certain shooting environment, and the design data-based image is an image of a door frame rendered from 3D cad data of a door frame of a corresponding vehicle type to match a corresponding shooting environment (e.g., shooting coordinates, shooting angle, etc.).

[0048] The reference image IMG2 may be an image for defining a feature point search area on the collected image IMG3. Here, the collected image IMG3 may be an image taken for at least a portion of the vehicle during the production process, and the feature point search area may be an area in a certain image determined to be used to search for feature points to be used in image matching. That is, in an embodiment, the feature point search area may include only an area in which the feature point detection performance in a certain image is determined to be high, while excluding an area in which the feature point detection performance in a certain image is determined to be low. Taking the door frame as an example, the reference image IMG2 may enable the area in which the feature point detection performance is determined to be high (for example, an area in which the image change according to the shooting environment is relatively small) and the area in which the feature point detection performance is determined to be low (for example, an area in which the image change according to the shooting environment is relatively large) to be distinguished on the collected image IMG3 for the door frame. The second image processing device 20 may define a feature point search area on the collected image IMG3 by using the reference image IMG2, and may perform image matching by detecting feature points from the feature point search area. In the present embodiment, it is shown that the mobile robot 30 shoots the captured image IMG1 and the collected image IMG3, but the scope of the present disclosure is not limited thereto.

[0049] The first image processing device 10 can, for example, receive a captured image IMG1 from the mobile robot 30, generate a reference image IMG2 for feature point search by utilizing a design data-based image that matches the captured image IMG1, and include a design data-based image generation module 100, an exclusion area determination module 110 and a reference image generation module 120.

[0050] The design data-based image generation module 100 can generate an image based on design data by matching the shooting coordinates corresponding to the shooting image IMG1 provided from the camera that shoots the vehicle (for example, the camera installed in the mobile robot 30) with the design data coordinates corresponding to the design data. For example, when the shooting coordinates of the door frame shot by the camera in a certain workspace are the coordinates x1, y1 and z1 on the first coordinate system, and the design data coordinates of the image rendered from the design data of the door frame are the coordinates x2, y2 and x2 on the second coordinate system, the image based on design data can be generated by matching (or converting) the shooting coordinates and the design data coordinates in the coordinates of the same coordinate system. For example, in the subsequent process, compared with the captured image captured by shooting the coordinates x1, y1 and z1 of the door frame of the actual vehicle on the first coordinate system, the image based on design data generated for the door frame can match the size, area, angle, etc. of the door frame actually seen. Of course, the method of matching the shooting image IMG1 with the image rendered from the design data by the design data-based image generation module 100 is not limited to the above method.

[0051] The exclusion area determination module 110 can distinguish between the vehicle body area and the component area in the design data-based image generated by the design data-based image generation module 100, and determine the partial area including the component area as the exclusion area. Here, when setting the feature point search area, the exclusion area can be used to indicate the area that needs to be excluded because the feature point detection performance is determined to be low. Specifically, when the exclusion area is determined in the exclusion area determination module 110, a mask area can be set based on these exclusion areas, and the work of only including the area in the feature point search area in the captured image where the feature point detection performance is determined to be high can be performed.

[0052] The exclusion area may include a parts area. The vehicle body area may include the vehicle body, and the parts area may include parts (e.g., cables) assembled to the vehicle body. However, since parts may be manually assembled by workers at any point in time in an unspecific fixed process sequence, when the parts area is included in the feature point search area and image matching is performed on captured images before and after assembly of the corresponding parts based on feature points detected in the corresponding parts area, the accuracy of image matching cannot be ensured. Since the parts area is included in the exclusion area, feature points with a high probability of reducing the accuracy of image matching can be prevented from being detected.

[0053] In some embodiments, the exclusion area determination module 110 can distinguish between the vehicle body area and the component area in the design data-based image automatically generated in the design data-based image generation module 100 by using the design data. Specifically, various values ​​can be assigned to the vehicle body or component identification number in the design data. For example, the design data corresponding to the identification number of the component corresponding to the vehicle body can be assigned a first value (e.g., "0") to indicate that the corresponding component is the vehicle body, and the design data corresponding to the identification number of the component corresponding to the component other than the vehicle body can be assigned a second value (e.g., "1") to indicate that the corresponding component is a component. The exclusion area determination module 110 can automatically distinguish between the vehicle body area and the component area based on the value assigned to the vehicle body or component identification number in the design data.

[0054] In some embodiments, the exclusion area may further include a background area. The background area may include, for example, an area excluding a main subject (e.g., a door frame) on the captured image IMG3 captured by the mobile robot 30, and may include, for example, other parts that are distinguished from the main subject, internal facilities of the factory, and areas where workers are photographed. When such a background area is included in the feature point search area, and image matching is performed on the captured image based on the feature points detected in the corresponding background area, the accuracy of the image matching cannot be ensured. Since the background area is included in the exclusion area, feature points with a high probability of reducing the accuracy of image matching can be prevented from being detected.

[0055] In some embodiments, the exclusion area may further include a user-specified area. The user-specified area may be an area specified by the user based on a large change in the design data image according to the environment. For example, the user may specify a portion of the reflective surface of the vehicle body that protrudes to have a high light reflection as an area based on the design data image that has a large change in the lighting environment inside the factory. When the user-specified area of ​​such a protruding portion is included in the feature point search area, and image matching is performed on the captured images before and after the light reflection occurs based on the feature points detected in the corresponding user-specified area, the accuracy of the image matching cannot be ensured. Since the user-specified area is included in the exclusion area, it is possible to prevent the detection of feature points that have a high probability of reducing the accuracy of the image matching.

[0056] The reference image generation module 120 may generate a reference image IMG2 including a masked area, which masks the exclusion area determined by the exclusion area determination module 110 in the design data image. The masked area may be generated by combining the exclusion areas. For example, according to the specific implementation purpose or environment, the masked area may include only the component area, may include the component area and the background area, may include the component area and the user-specified area, or may include all of the component area, the background area, and the user-specified area. The generated reference image IMG2 may be sent to the second image processing device 20.

[0057] In some embodiments, the reference image generation module 120 can display the masked area by using binary values. For example, the reference image IMG2 can be an image of a predetermined size, and the masked area and other areas can be distinguished by the binary values ​​assigned to the individual pixels constituting the corresponding image. For example, a first binary value (e.g., "0") can be assigned to the pixels corresponding to the masked area, and a second binary value (e.g., "1") can be assigned to the pixels corresponding to the other areas, and the representation scheme of the masked area is not limited to the above method.

[0058] The second image processing device 20 can, for example, receive a captured image IMG3 taken by the mobile robot 30, can perform image matching on the captured image IMG3 based on a feature point search area defined in a reference image IMG2, and can include a feature point search area definition module 200, a feature point detection module 210 and an image matching module 220.

[0059] The feature point search area definition module 200 may define a feature point search area in the acquisition image IMG3 by using the reference image IMG2 received from the reference image generation module 120. Specifically, the feature point search area definition module 200 may overlap the reference image IMG2 including the mask area on the acquisition image IMG3, exclude the area on the acquisition image IMG3 covered by the mask area from the feature point search area, and include the area on the acquisition image IMG3 not covered by the mask area in the feature point search area.

[0060] In some embodiments, the reference image IMG2 may be generated in the same size as the acquisition image IMG3 at a time point generated by the reference image generation module 120. In this case, the feature point search area definition module 200 may overlap the acquisition image IMG3 with the reference image IMG2 without separate resizing work.

[0061] In some embodiments, the sizes of the reference image IMG2 and the acquired image IMG3 received by the feature point search area definition module 200 may be different. In this case, the feature point search area definition module 200 may perform resizing so that the size of the reference image IMG2 matches the size of the acquired image IMG3, or vice versa, perform resizing so that the acquired image IMG3 matches the size of the reference image IMG2, and then the acquired image IMG3 may overlap with the reference image IMG2. In addition, when necessary, image cropping, rotation, etc. may be selectively performed.

[0062] In some embodiments, the reference image IMG2 generated by the reference image generation module 120 can be stored in a database, and when a new acquisition image IMG3 is generated, the feature point search area definition module 200 can obtain the reference image IMG2 corresponding to the acquisition image IMG3 from the database. At this time, the reference image IMG2 stored in the database can be appropriately marked for each search based on the acquisition image IMG3.

[0063] The feature point detection module 210 may detect a plurality of feature points in the feature point search area defined by the feature point search area definition module 200. The feature points detected in the feature point search area defined by the feature point search area definition module 200 may be feature points that can be referenced for accurate image matching even if the shooting environment changes.

[0064] In some embodiments, the feature point detection module 210 can perform histogram matching on the collected image IMG3, perform correction on the brightness and contrast in the feature point search area, adjust so that the illumination similarity becomes higher, and then detect multiple feature points. For example, based on the histogram values ​​pre-obtained for the feature point search area related to the captured image IMG1 or the reference image IMG2, the histogram values ​​in the feature point search area of ​​the collected image IMG3 are corrected, thereby further improving the performance of feature point detection.

[0065] The image matching module 220 can perform image matching on the captured image IMG3 based on the multiple feature points detected by the feature point detection module 210. When the image matching result does not exceed the error range of the capture device (e.g., the mobile robot 30), the image matching can be terminated, and when the image matching result exceeds the error range of the capture device, feature points can be reselected to re-execute the image matching. The operation of re-executing the image matching according to the error range will be described later. Figures 10 to 17 Give a description.

[0066] According to the present embodiment, by excluding from the feature point search area an area where the image changes relatively greatly due to environmental changes such as uncontrollable lighting, or an area where there is an image phase change between when parts are manually assembled and when parts are not assembled, the detection rate and accuracy of feature points for image matching between acquired images can be improved, thereby improving the performance of detecting feature points used in image matching. In addition, since the exclusion area in the feature point search area of ​​the acquired image can be automatically determined based on design data (e.g., 3D cad data), the feature point search area can be effectively and efficiently set in a short period of time. Therefore, in industrial sites where it is difficult to control working environments such as lighting, the performance of image matching can be improved, and the image acquisition and matching process can be improved to have an automated solution, which enables the visual system to operate effectively while reducing the number of personnel such as visual experts and engineers in the manufacturing plant.

[0067] In the following, reference is made to Figures 2 to 8 , using a pillar supported between a door and a roof lining of a vehicle as an example, the operation of the above-mentioned image processing system 1 will be described.

[0068] Figure 2 FIG. 1 is a diagram for explaining an image based on design data that can be used in the image processing system according to the embodiment.

[0069] Reference Figure 2The design data-based pillar image generated based on the 3D cad data for the pillar of the vehicle is a pillar image that is pre-rendered to match the shooting environment (e.g., shooting coordinates in the coordinate system related to the mobile robot 30, shooting angle, etc.) of the pillar image shot by the mobile robot 30. For example, the design data-based pillar image generated based on the 3D cad data for the pillar can be generated by matching the shooting coordinates corresponding to the shot pillar image provided by the mobile robot 30 that shoots the pillar and the design data coordinates based on the 3D cad data for the pillar in the coordinates of the same coordinate system. Figure 2 Image based on the design data pillar.

[0070] Figures 3 to 5 FIG. 1 is a diagram for explaining determination of an exclusion area in the image processing system according to the embodiment.

[0071] Reference Figure 3 ,express Figure 2 A component region M1 distinguished from the vehicle body region in the pillar image based on the design data. Since the components assembled to the pillar, for example, the cable, can be assembled manually by a worker at any time point in accordance with an unspecific fixed process sequence, when the component region M1 including the cable is included in the feature point search region, and image matching is performed on the captured images before and after the cable assembly based on the feature points detected in the component region M1, the accuracy of the image matching cannot be ensured. Therefore, the component region M1 can be included in the exclusion region.

[0072] Reference Figure 4 ,express Figure 2 The background area M2 that is distinguished from the vehicle body area in the pillar image based on the design data. The feature point search area may include other parts that are distinguished from the main body pillar, the internal facilities of the factory, or the background area M2 where the workers are photographed, and when image matching is performed on the captured image based on the feature points detected in the background area M2 for the captured image with different backgrounds, the accuracy of the image matching cannot be guaranteed. Therefore, the background area M2 can be included in the exclusion area.

[0073] Reference Figure 5 ,express Figure 2 The user-specified area M3 that is distinguished from the vehicle body area in the design data pillar image. The user can designate the portion of the vehicle body that highlights the reflective surface to have high light reflection, that is, the area that varies greatly according to the lighting environment inside the factory based on the design data image, as the user-specified area M3, which may be included in the feature point search area, and when image matching is performed on the captured images with strong light reflection and without light reflection based on the feature points detected in the user-specified area M3, the accuracy of the image matching cannot be ensured. Therefore, the user-specified area M3 can be included in the exclusion area.

[0074] Figure 6 is a diagram for explaining a reference image that can be used in the image processing system according to the embodiment.

[0075] Reference Figure 6 , the reference pillar image may include corresponding to Figure 3 The masking area E1 of the component area M1 shown in FIG. Figure 4 The masked areas E2 and E3 of the background area M2 shown in FIG. Figure 5 The masked area E4 of the user specified area M3 shown in . In this embodiment, the reference pillar image is shown to include all of the masked areas E1, E2, E3 and E4, but depending on the specific implementation purpose or environment, only some of the masked areas E1, E2, E3 and E4 may be included.

[0076] Figure 7 and Figure 8 : is a diagram for explaining a feature point search area defined in a captured image by using a reference image in the image processing system according to the embodiment.

[0077] Reference Figure 7 , the captured pillar image shown is an image taken during the production process by the mobile robot 30. Since the shooting is performed during the production process, it can be seen that the interior of the factory behind the pillar is also photographed.

[0078] Reference Figure 8 , the reference pillar image including the masked areas E1, E2, E3 and E4 can be overlapped on the acquired pillar image, the area covered by the masked areas E1, E2, E3 and E4 on the acquired pillar image can be excluded from the feature point search area, and the area not covered by the masked areas E1, E2, E3 and E4 on the acquired pillar image can be included in the feature point search area. A plurality of feature points P can be detected in the feature point search area thus defined.

[0079] Fig. 9 It is a diagram for explaining the image processing method according to the embodiment.

[0080] Reference Fig. 9According to the image processing method of the embodiment, in step S901, the image based on the design data is generated by matching the shooting coordinates corresponding to the shooting image with the design data coordinates corresponding to the design data, and in step S903, the body area and the parts area in the design data image are distinguished, and the partial area including the parts area is determined as the exclusion area. In addition, the method can generate a reference image including a masking area that masks the exclusion area in the design data image in step S905, and the method can define a feature point search area in the captured image by using the reference image in step S907. In addition, the method can detect multiple feature points in the feature point search area in step S909.

[0081] For further details of the image processing method according to this embodiment, please refer to Figures 1 to 8 The description will not be repeated here.

[0082] Fig.10 is a diagram for explaining an image processing system according to an embodiment.

[0083] Reference Fig.10 According to the embodiment, the image processing system 2 may include a first image processing device 10, a second image processing device 20 and a mobile robot 30. Figure 1 The described image processing system 1 may differ in that the second image processing device 20 further comprises a transformation matrix generation module 230 and an error analysis module 240 .

[0084] When performing image matching, the transformation matrix generation module 230 can generate a transformation matrix based on the position deviation from the previously acquired acquisition image. The transformation matrix can be generated when the position deviation from the existing image is calculated after the image is acquired, and can include information about motion (i.e., translation), scaling, shearing, rotation, and tilt. For example, the transformation matrix can be defined as a 3×3 matrix as follows:

[0085] |Sx Shy E|

[0086] |Shx Sy F|

[0087] |Tx Ty 1|

[0088] Here, Tx and Ty may represent displacements along the x-axis and y-axis, respectively, and Sx and Sy may represent scaling factors along the x-axis and y-axis, respectively. Shx and Shy may represent shear factors along the x-axis and y-axis, respectively. E and F may be factors affecting the vanishing point with respect to tilt. In some embodiments, the transformation matrix may be configured to include cos(q), sin(q), -sin(q), cos(q) values ​​of a rotation angle q about a certain center, instead of Tx, Ty, Sx, Sy, Shx, Shy, E, and F.

[0089] The error analysis module 240 may analyze whether the component values ​​of the transformation matrix correspond to a predefined error range. Here, the predefined error range may include an error range for each image acquisition device, such as a shooting position of the mobile robot 30. The mobile robot 30 may repeatedly perform image acquisition at each shooting position, and may calculate the error range of each shooting position with respect to each of the components of the transformation matrix.

[0090] If the component values ​​of the transformation matrix generated according to the image matching are outside the error range, the image matching module 220 may reselect feature points and re-perform the image matching.

[0091] Fig.11 and Fig.12 A diagram for explaining the error range of image acquisition due to errors in the acquisition equipment.

[0092] Reference Fig.11 , the mobile robot 30 can repeatedly perform image acquisition at various shooting positions about the vehicle. For example, the mobile robot 30 can perform image acquisition at the first position P20102, the second position P20204, the third position P20302, the fourth position P20402, the fifth position P20502 and the sixth position P20603, and refer to Fig.12 , for each of the components of the transformation matrix, the error range calculated for each shooting position can be known. Specifically, for each Sx, Sy, Shx, Shy component of the transformation matrix, it can be seen that the error range is calculated for each of the positions P20102 to P20603. For example, in the case where the Sx component of the transformation matrix is ​​generated when performing image matching, the error analysis module 240 can analyze whether the value of the Sx component corresponds to an error range of about 0.8 to about 1.05 with respect to the third position P20302, and when the value of the Sx component of the transformation matrix generated when performing image matching exceeds the error range of about 0.8 to about 1.05, the image matching module 220 can reselect feature points and re-perform image matching.

[0093] In the following, reference will be made to Figures 13 to 17An embodiment of re-performing image matching is described in detail.

[0094] Fig.13 and Fig.14 It is a diagram for explaining feature point detection when the error range is exceeded.

[0095] Reference Fig.13 , the acquired image may overlap with the reference image including the masked areas E5, E6, E7 and E8, the areas on the acquired image covered by the masked areas E5, E6, E7 and E8 may be excluded from the feature point search area, and the areas on the acquired image not covered by the masked areas E5, E6, E7 and E8 may be included in the feature point search area.

[0096] The feature point detection module 210 can be Fig.13 In the feature point search area shown, n (n is a natural number, for example, 10) feature points are detected, and the image matching module 220 can perform image matching on the captured image based on m (m is a natural number less than n, for example, 3) feature points P1, P2 and P3 initially selected from the plurality of n feature points. In addition, when performing image matching, the transformation matrix generation module 230 can generate a transformation matrix based on the position deviation from the previously captured image, and the error analysis module 240 can analyze whether the component values ​​of the transformation matrix correspond to a predefined error range.

[0097] Reference Fig.14 , when the component values ​​of the transformation matrix generated according to the image matching using the m feature points P1, P2 and P3 exceed the error range, the image matching module 220 may reselect the m feature points P4, P5 and P6 from the n feature points for a second time and re-execute the image matching. Thereafter, when re-executing the image matching, the transformation matrix generation module 230 may re-generate the transformation matrix according to the position deviation from the previously acquired acquisition image, and the error analysis module 240 may analyze whether the component values ​​of the transformation matrix correspond to the predefined error range.

[0098] Fig.15 It is a diagram for explaining the image processing method according to the embodiment.

[0099] Reference Fig.15According to the image processing method of the embodiment, in step S1501, a feature point search area may be defined in the acquired image by using a reference image, and in step S1503, a plurality of feature points including a first feature point may be detected in the feature point search area. In addition, the method may perform image matching on the acquired image based on the plurality of feature points including the first feature point in step S1505, and the method may generate a transformation matrix according to a position deviation from a previously acquired acquired image when performing image matching in step S1507. In addition, the method may determine whether the component values ​​of the transformation matrix correspond to a predefined error range in step S1509.

[0100] When it is determined that the component values ​​of the transformation matrix correspond to the predetermined error range, the image matching can be successfully terminated, and when it is determined that the component values ​​of the transformation matrix exceed the predetermined error range, the method can proceed to step S1503 to reselect multiple feature points including a second feature point different from the first feature point. Thereafter, in step S1505, image matching can be performed on the captured image based on the multiple feature points including the second feature point.

[0101] For more details of the image processing method according to this embodiment, please refer to Figures 1 to 14 The description is not repeated here.

[0102] Fig.16 It is a diagram for explaining feature point detection when the error range is exceeded.

[0103] Reference Fig.16 , the acquired image may overlap with the reference image including the masked areas E5, E6, E7 and E8, the areas on the acquired image covered by the masked areas E5, E6, E7 and E8 may be excluded from the feature point search area, and the areas on the acquired image not covered by the masked areas E5, E6, E7 and E8 may be included in the feature point search area.

[0104] The feature point detection module 210 can be Fig.16 A plurality of feature points are detected in the feature point search area shown, and the image matching module 220 can perform image matching on the acquired image based on the plurality of feature points. In addition, when performing image matching, the transformation matrix generation module 230 can generate a transformation matrix according to the position deviation from the previously acquired acquired image, and the error analysis module 240 can analyze whether the component values ​​of the transformation matrix correspond to a predefined error range.

[0105] When the component values ​​of the transformation matrix exceed the predefined error range, the feature point search area definition module 200 may increase or decrease at least a portion A-A' of the boundary line of the feature point search area in units of pixels to change it into a feature point search area including a new boundary line B-B'. Thereafter, the feature point detection module 210 may detect new feature points in the changed feature point search area, and the image matching module 220 may perform image matching on the acquired image based on the new feature points.

[0106] Fig.17 It is a diagram for explaining the image processing method according to the embodiment.

[0107] Reference Fig.17 According to the image processing method of the embodiment, in step S1701, a feature point search area including a first boundary line may be defined in the acquired image by using a reference image, and in step S1703, a plurality of feature points may be detected in the feature point search area. In addition, the method may perform image matching on the acquired image based on the plurality of feature points in step S1705, and the method may generate a transformation matrix according to a position deviation from a previously acquired acquired image when performing image matching in step S1707. In addition, the method may determine whether the component values ​​of the transformation matrix correspond to a predefined error range in step S1709.

[0108] When it is determined that the component value of the transformation matrix corresponds to the predefined error range, the image matching can be successfully terminated, and when it is determined that the component value of the transformation matrix exceeds the predefined error range, the method can proceed to step S1701 to define a feature point search area including a new second boundary line obtained by increasing or decreasing the first boundary line in units of pixels. Thereafter, in steps S1703 and S1705, new feature points can be detected in the changed feature point search area, and image matching can be performed on the captured image based on the new feature points.

[0109] For more details of the image processing method according to this embodiment, please refer to Figures 1 to 12 and Fig.16 The description is not repeated here.

[0110] In some embodiments, when the error range is exceeded, the above combination Figures 13 to 15 The described embodiments are combined with the above Fig.16 and Fig.17 The described embodiments may be implemented in combination.

[0111] For example, the feature point detection module 210 may detect n (n is a natural number, for example, 10) feature points in a feature point search area including the first boundary line, and the image matching module 220 may perform image matching on the captured image based on m (m is a natural number less than n, for example, 3) feature points initially selected from the plurality of n feature points. In addition, when performing image matching, the transformation matrix generation module 230 may generate a transformation matrix based on a positional deviation from a previously captured image, and the error analysis module 240 may analyze whether the component values ​​of the transformation matrix correspond to a predefined error range.

[0112] When the component values ​​of the transformation matrix generated according to the image matching using the m feature points are out of the error range, the image matching module 220 may reselect m feature points from among the n feature points for a second time and re-perform the image matching.

[0113] When the component values ​​of the transformation matrix still exceed the error range even after the image matching module 220 repeats the process of reselecting m feature points from among n feature points and re-performing image matching a predetermined number of times (e.g., 5 times) or more, the feature point search area definition module 200 may define a feature point search area including a new second boundary line obtained by increasing or decreasing the first boundary line in units of pixels. Thereafter, n feature points may be detected again in the changed feature point search area, and the process of performing image matching on the captured image based on the m feature points initially selected from among the plurality of n feature points may be repeated.

[0114] In addition, in some embodiments, the first boundary line can be set to enable an increase or decrease by a predetermined pixel value (for example, 5 pixels), and when the increase or decrease value reaches the predetermined pixel value, an increase or decrease in pixels can be performed for boundary lines other than the first boundary line in the feature point search area including the first boundary line.

[0115] According to the above-mentioned embodiment, by excluding from the feature point search area an area where the image changes relatively greatly due to environmental changes such as uncontrollable lighting, or an area where there is an image phase change between when parts are manually assembled and when parts are not assembled, the detection rate and accuracy of feature points for image matching between acquired images can be improved, thereby improving the performance of detecting feature points used in image matching. In addition, since the exclusion area in the feature point search area of ​​the acquired image can be automatically determined based on design data (e.g., 3D cad data), the feature point search area can be effectively and efficiently set in a short period of time. Therefore, in industrial sites where it is difficult to control working environments such as lighting, the performance of image matching can be improved, and the image acquisition and matching process can be improved to have an automated solution, which enables the visual system to operate effectively while reducing the number of personnel such as visual experts and engineers in the manufacturing plant.

[0116] Fig.18 is a diagram for explaining an embodiment of a computing device for realizing an image processing apparatus, an image processing system, and an image processing method according to the embodiment.

[0117] Reference Fig.18 , the image processing apparatus, the image processing system, and the image processing method according to the embodiment can be implemented by using the computing device 500.

[0118] The computing device 500 may include at least one of a processor 510, a memory 530, a user interface input device 540, a user interface output device 550, and a storage device 560 that communicate with each other through a bus 520. The computing device 500 may include a network interface 570 electrically connected to the network 40. The network interface 570 may send or receive signals to or from other entities through the network 40.

[0119] The processor 510 may be implemented in various types such as a microcontroller unit (MCU), an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), etc., and may be any semiconductor device configured to execute instructions stored in the memory 530 or the storage device 560. The processor 510 may be configured to implement the above-mentioned Figures 1 to 17 Describes the functions and methods.

[0120] The memory 530 and the storage device 560 may include various types of volatile or non-volatile storage media. For example, the memory may include a read-only memory (ROM) 531 and a random access memory (RAM) 532. In the present embodiment, the memory 530 may be located inside or outside the processor 510, and the memory 530 may be connected to the processor 510 in various known ways.

[0121] In some exemplary embodiments, at least some of the configurations or functions of the image processing apparatus, image processing system, and image processing method according to the exemplary embodiments may be implemented by a program or software executed by the computing device 500, and the program or software may be stored in a computer-readable medium.

[0122] In some embodiments, at least part of the configuration or function of the image processing apparatus, the image processing system, and the image processing method according to the embodiments may be implemented as hardware that may be electrically connected to the computing device 500 .

[0123] While the disclosure has been described in connection with what are presently considered to be practical embodiments, it will be understood that the disclosure is not limited to the disclosed embodiments, but on the contrary is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An image processing device, comprising: An exclusion area determination module is configured to distinguish between a vehicle body area and a component area in the design data image, and determine a partial area including the component area as an exclusion area; A reference image generating module, configured to generate a reference image, wherein the reference image includes a masked area, and the masked area masks the excluded area in the design data-based image; a feature point search area definition module, configured to define a feature point search area in the acquired image by using the reference image; as well as The feature point detection module is configured to detect a plurality of feature points in the feature point search area.

2. The image processing device according to claim 1, further comprising: The design data-based image generation module is configured to generate the design data-based image by matching shooting coordinates with design data coordinates, wherein the shooting coordinates correspond to the shot image provided from a camera, the camera is configured to shoot the vehicle, and the design data coordinates correspond to the design data.

3. The image processing apparatus according to claim 1, wherein: The exclusion region determination module is configured to distinguish the vehicle body region from the component region according to a value assigned to each vehicle body or component identification number in the design data.

4. The image processing apparatus according to claim 1, wherein: The exclusion area further includes a background area, and the background area is distinguished from the vehicle body area and the component area in the design data-based image.

5. The image processing apparatus according to claim 1, wherein: The exclusion area further includes a user-specified area, and the user-specified area is an area in the design data-based image that is specified by a user and has a large change depending on the environment.

6. The image processing device according to claim 1, further comprising: The image matching module is configured to perform image matching on the collected image based on the multiple feature points.

7. The image processing device according to claim 6, further comprising: a transformation matrix generation module configured to generate a transformation matrix according to a position deviation from a previously acquired captured image when performing the image matching; as well as The error analysis module is configured to analyze whether the component values ​​of the transformation matrix correspond to a predefined error range.

8. The image processing apparatus according to claim 7, wherein: The plurality of feature points include a first feature point and a second feature point different from the first feature point; and The image matching module is configured to perform the image matching on the acquired image by using the second feature points when a component value of a transformation matrix generated according to the image matching using the first feature points exceeds the error range.

9. The image processing apparatus according to claim 7, wherein: The feature point search area definition module is configured to change the feature point search area by increasing or decreasing at least a portion of a boundary line of the feature point search area in units of pixels when a component value of the transformation matrix exceeds a predefined error range; The feature point detection module is configured to detect new feature points in the changed feature point search area; and The image matching module is configured to perform the image matching on the acquired image based on the new feature points.

10. An image processing system, comprising: a mobile robot configured to photograph the vehicle at a photographing coordinate determined in a pre-provided teaching; a first image processing device that receives a first captured image from the mobile robot and is configured to generate a reference image for feature point search by using a design data-based image matched with the first captured image; as well as The second image processing device receives a second captured image from the mobile robot and is configured to perform image matching on the second captured image based on a feature point search area defined in the reference image.

11. The image processing system according to claim 10, wherein: The first image processing device comprises: an exclusion area determination module, configured to distinguish the vehicle body area and the component area in the image based on the design data, and determine a partial area including the component area as an exclusion area; and The reference image generation module is configured to generate the reference image, wherein the reference image includes a masked area, and the masked area masks the excluded area in the design data-based image.

12. The image processing system according to claim 11, wherein: The exclusion region determination module is configured to distinguish the vehicle body region from the component region according to a value assigned to each vehicle body or component identification number in the design data.

13. The image processing system according to claim 11, wherein: The second image processing device comprises: a feature point search area definition module configured to define the feature point search area in the second captured image by using the reference image; and The feature point detection module is configured to detect a plurality of feature points in the feature point search area.

14. The image processing system according to claim 13, wherein: The second image processing device further comprises: an image matching module, configured to perform the image matching on the second captured image based on the plurality of feature points; a transformation matrix generation module configured to generate a transformation matrix according to a position deviation from a previously acquired second captured image when performing the image matching; and The error analysis module is configured to analyze whether the component values ​​of the transformation matrix correspond to a predefined error range.

15. The image processing system according to claim 14, wherein: The plurality of feature points include a first feature point and a second feature point different from the first feature point; and The image matching module is configured to perform the image matching on the second photographed image by using the second feature points when a component value of a transformation matrix generated according to the image matching using the first feature points exceeds the error range.

16. The image processing system according to claim 14, wherein: The feature point search area definition module is configured to change the feature point search area by increasing or decreasing at least a portion of a boundary line of the feature point search area in units of pixels when a component value of the transformation matrix exceeds the predefined error range; The feature point detection module is configured to detect new feature points in the changed feature point search area; and The image matching module is configured to perform the image matching on the second photographed image based on the new feature points.

17. An image processing method, comprising: Distinguish between body area and component area in the image based on design data; determining a partial area including the component area as an exclusion area; generating a reference image, wherein the reference image includes a masked area, and the masked area masks the excluded area in the design data-based image; defining a feature point search area in the acquired image by using the reference image; as well as A plurality of feature points are detected in the feature point search area.

18. The image processing method according to claim 17, further comprising: performing image matching on the acquired image based on the plurality of feature points; When performing the image matching, generating a transformation matrix based on the position deviation from the previously acquired captured image; as well as It is analyzed whether the component values ​​of the transformation matrix correspond to a predefined error range.

19. The image processing method according to claim 18, wherein: The plurality of feature points include a first feature point and a second feature point different from the first feature point; and The image processing method further includes: when a component value of a transformation matrix generated according to the image matching using the first feature point exceeds the error range, performing the image matching on the acquired image by using the second feature point.

20. The image processing method according to claim 18, further comprising: When the component value of the transformation matrix exceeds the predefined error range, changing the feature point search area by increasing or decreasing at least a portion of a boundary line of the feature point search area in units of pixels; Detecting new feature points in the changed feature point search area; as well as The image matching is performed on the acquired image based on the new feature points.

Images