Visual inspection system and method using mobile robot

Abstract

Disclosed is a visual inspection system using a mobile robot. A visual inspection system using a mobile robot according to an embodiment of the present invention may comprise: a mobile robot configured to move to at least one specified inspection position (P) and to photograph an inspection area of a component assembled with a product; and a detection server configured to obtain a detection image captured by the mobile robot, automatically calculate an image deviation between the detection image and a reference image, and evaluate the assembly quality of the detection image at each detection position through a detection algorithm, an image deviation reflecting a repetitive position error of the robot for each detection position is detected by a detection algorithm learned by performing image enhancement reflecting a range of the image deviation.

Description

Technical Field

[0001] The present disclosure relates to a vision inspection system and method using a mobile robot, and more particularly, to a vision inspection system and method for real-time inspection of the manual assembly quality of workers in a vehicle assembly process using a mobile robot. Background Art

[0002] Generally, an automobile manufacturing plant assembles a vehicle body and components through various processes. In the manual assembly process of workers, real-time quality inspection is required to ensure product quality.

[0003] Conventional real-time quality inspection uses a camera mounted on an articulated fixed robot to capture inspection images for performing vision inspection. However, conventional vision inspection methods have limitations in controlling the shooting posture (attitude) and shooting area of the fixed robot. Therefore, multiple robots must be operated, and the working paths of workers and robots overlap, which hinders work or poses a collision risk.

[0004] Therefore, recently, methods of mounting a camera on a mobile robot and capturing inspection images without disturbing the working path of workers have been explored. However, when using a mobile robot, there is a problem that due to the position error (shooting deviation) during camera shooting, image deviation occurs repeatedly every time an image is obtained, which results in a decline in vision inspection performance.

[0005] For example, even if the mobile robot is set to capture the inspection area of a vehicle from a specified inspection position, due to its movable characteristics, there is a deviation between the inspection images captured each time due to the minute repeated position error of the mobile robot. In addition, the position error of these robots causes a deviation between the reference image of the component set for assembly quality inspection and the inspection image.

[0006] The deviation of the inspection images using these mobile robots results in a decline in the quality inspection performance of the vision inspection system. Therefore, improvement measures are needed to improve the inspection performance.

[0007] The above information disclosed in this background art section is only used to enhance the understanding of the background of the present disclosure. Therefore, it may include information that does not constitute the prior art known to those of ordinary skill in the art in the country. Summary of the Invention

[0008] Technical Problem

[0009] The present disclosure attempts to provide a vision inspection system and method using a mobile robot, which can automatically calculate the image deviation reflecting the repeated position error of the robot when performing vision inspection using a mobile robot in an industrial field, and perform vision inspection on each inspected component by constructing an inspection algorithm reflecting the image deviation.

[0010] The present disclosure also attempts to provide a vision detection system using a mobile robot, which can automatically replace an existing detection algorithm with a new detection algorithm generated by automatically specifying an enhancement range reflecting image deviations caused by various environmental changes, and maintain the best detection quality in real time.

[0011] Technical solution

[0012] A vision detection system using a mobile robot may include: a mobile robot configured to move to at least one specified detection position (P) and capture a detection area of components assembled with a product; and a detection server configured to obtain a detection image captured by the mobile robot, calculate an image deviation of the detection image compared with a reference image, and evaluate the assembly quality of the detection image at each detection position through a detection algorithm, where the image deviation reflects the repetitive position error of the robot for each detection position, and the detection algorithm is learned by performing image enhancement reflecting the range of the image deviation.

[0013] The mobile robot may include: a vision sensor module configured to generate a detection image captured at the detection position; an autonomous driving sensor module configured to detect the surrounding environment through sensors; a mobile module configured to move freely by driving wheels or quadruped walking; a wireless communication module configured to wirelessly transmit the captured detection image to the detection server; and a control module configured to perform control to identify the position of a worker through the vision sensor module, follow the worker to move to the detection position (P) where the work has been completed, and capture a detection image through the vision sensor module.

[0014] The control module may store the assembly positions of the components to be assembled in the corresponding process according to the type and specifications of the product, and at least one detection position specified for capturing the detection image of the components.

[0015] The control module may be configured to distinguish multiple regions divided around the product and restrict movement so as not to enter the worker area where the worker is currently located.

[0016] The vision sensor module may be mounted on the mobile robot through a robotic arm and configured to capture detection images of external and internal components of the product through attitude control of the robotic arm.

[0017] The detection server may include: a communication unit configured to obtain a detection image from a mobile robot; an image processing unit configured to store a reference image corresponding to the detection image of each component part and generate a corrected image converted by performing a matching process on the detection image by comparing it with the reference image; a deep learning unit configured to construct a detection algorithm for each detected component part of a corresponding process by pre-enhancing the detection image and learning the range of repeated position errors at each detected position of the mobile robot through deep learning; a database (DB) configured to store programs and data for operating the detection server; and a controller configured to generate a new detection algorithm by specifying an image enhancement range for deep learning based on the transformation matrix obtained during the image matching process.

[0018] The image processing unit may be configured to calculate an image matching error of the detection image with respect to the reference image during the image matching process and calculate an image deviation between the two images to extract transformation matrix information that can be obtained when correcting the detection image.

[0019] The transformation matrix information may include information on translation for image movement, rotation for image rotation, scaling for image enlargement / reduction, tilt for image slope conversion, and shear for image shearing as values for quantifying the image deviation.

[0020] The image processing unit may be configured to extract at least one detected component part image corresponding to the detected component part region (ROI) set for the reference image in the corrected image as learning data.

[0021] The deep learning unit may be configured to perform image enhancement using the transformation matrix range and the detected component part images extracted by the image processing unit and enhance the learning images reflecting the range of repeated position errors of the corresponding detected component parts.

[0022] The deep learning unit may be configured to learn the detected component part images through deep learning by the detection algorithm of the corresponding component part and output one of the detection results of good (OK), defective (NG), and detection error (NA).

[0023] The controller may be configured to generate a new detection algorithm that reflects the image enhancement range of deep learning in real time based on the transformation matrix information obtained during the image matching process of the detection image.

[0024] The controller may be configured to replace or update the existing detection algorithm with the new detection algorithm when it is determined that the performance has been improved by comparing the detection result obtained by re-learning through deep learning using the new detection algorithm with the detection result obtained by using the existing detection algorithm.

[0025] A vision detection method using a mobile robot for real-time detection of the manual assembly quality of workers in the vehicle assembly process. The vision detection method includes: obtaining a detection image captured at a specific detection position around the product vehicle by using the mobile robot; extracting a detected component image from a corrected image, which is converted by performing image matching processing on the detection image to match a preset reference image; automatically calculating an image deviation reflecting the repeated position error of the robot for each detection position by comparing the detection image with the reference image, storing a detection algorithm learned by performing image enhancement within the range reflecting the image deviation, and performing vision detection through deep learning of the detection algorithm corresponding to the detected component image of the detected component; and obtaining one detection result among Good (OK), Defect (NG), and Detection Error (NA) based on the vision detection.

[0026] Obtaining the detection result further includes: when the detection result is determined to be Defect (NG) or Detection Error (NA), extracting the misdetected or undetermined image through the operator's confirmation of the detection result, and storing the extracted image as evaluation data for confirming the performance of the new detection algorithm.

[0027] Extracting the detected component image may further include: generating a new detection algorithm by specifying the image enhancement range of deep learning based on the transformation matrix obtained during the image matching process.

[0028] Generating the new detection algorithm may include: calculating the image matching error of the detection image relative to the reference image; extracting a transformation matrix by calculating the image deviation between the two images, and the transformation matrix can be obtained when correcting the detection image; calculating the distribution of the transformation matrix values corresponding to the detection positions and storing the range in the DB; specifying the transformation matrix range of image enhancement during the deep learning of the corresponding detected component image based on the repeated position error range stored in the DB; and deep learning multiple learning images enhanced by reflecting the repeated position error of the mobile robot to generate a new detection algorithm for the corresponding component, and re-learning the detected component image through deep learning.

[0029] Specifying the matrix range can use enhancement during deep learning based on the repeated position error range of the transformation matrix to specify the range of random number generation for the translation, rotation, scaling, inclination, and shear conversion values of the detected component image.

[0030] Specifying the matrix range can generate random numbers according to the Gaussian normal distribution and perform deep learning by assigning weight values to the image deviation with a high probability of generating the repeated error of the mobile robot according to the position.

[0031] The visual detection method may further include: after re-learning through deep learning, when it is determined that the performance has improved by comparing the results of the re-learning with the evaluation data, replacing or updating the existing detection algorithm with a new detection algorithm.

[0032] Advantageous Effects

[0033] According to an embodiment, when performing visual detection using a mobile robot, by automatically calculating the image deviation reflecting the repeated position error of the mobile robot and performing visual detection by generating a detection algorithm reflecting the image deviation, the detection performance for the repeated position error of the mobile robot can be improved.

[0034] In addition, when performing visual detection, by specifying the enhancement range of the image deviation caused by the aging of the device over time or various environmental changes to generate a new detection algorithm and replacing / updating the existing detection algorithm with the new detection algorithm, the degradation of the detection performance can be prevented, and the optimal detection quality can be maintained in real time.

[0035] In addition, the mobile robot can follow the worker to perform visual detection and restrict its movement so that it does not enter the area where the worker is currently located, thereby preventing collisions without disturbing the worker's work path. Description of the Drawings

[0036] Figure 1 Shows a visual detection system using a mobile robot applied to a product assembly process according to an embodiment.

[0037] Figure 2 Is a block diagram schematically showing the configuration of a visual detection system using a mobile robot according to an embodiment.

[0038] Figure 3 Is a diagram for explaining the problem of image deviation occurring during visual detection using a mobile robot according to an embodiment.

[0039] Figure 4 Represents a detection image processing method for visual detection according to an embodiment.

[0040] Figure 5 Represents a transformation matrix according to an embodiment.

[0041] Figure 6 Represents a method for constructing a detection algorithm for each detection component according to an embodiment.

[0042] Figure 7 Represents a visual detection method using the detection algorithm of each detection component according to an embodiment.

[0043] Figure 8It is a flowchart schematically showing a vision detection method using a mobile robot according to an embodiment. Detailed implementation

[0044] The present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the present disclosure are shown.

[0045] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the present disclosure. As used herein, unless the context clearly dictates otherwise, the singular forms are also intended to include the plural forms. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated items.

[0046] Throughout the specification, terms such as first, second, "A", "B", "(a)", "(b)", etc. will only be used to describe various elements and will not be construed as limiting these elements. These terms are only used to distinguish the constituent elements from other constituent elements, and the nature or order of the constituent elements is not limited by these terms.

[0047] In this specification, it will be understood that when a component is referred to as being "connected" or "coupled" to another component, the one component can be directly connected or coupled to the other component, or connected or coupled to the other component through yet another component therebetween. In this specification, it will be understood that when a component is referred to as being "directly connected or coupled" to another component, the one component can be connected or coupled to the other component without yet another component therebetween.

[0048] In addition, it is understood that one or more of the following methods or aspects thereof can be executed by at least one controller. The term "controller" may refer to a hardware device including a memory and a processor. The memory is configured to store program instructions, and the processor is specifically programmed to execute the program instructions to perform one or more processes described further below. As described herein, the controller can control the operation of units, modules, components, devices, etc. Further, it is understood that, as will be understood by those of ordinary skill in the art, the following methods can be executed by a device including a controller together with one or more other components.

[0049] In addition, throughout the specification, various terms related to "image" are used, which are defined as follows.

[0050] The term "detection image" may refer to the original image obtained when a mobile robot captures a detection area at a specific detection position (e.g., P1, P2, ……, Pn) during actual visual inspection.

[0051] The term "reference image" is the best image obtained by capturing the detection area after moving the mobile robot to a pre-specified detection position, e.g., P1, P2, ……, Pn, and can serve as a reference for evaluating the assembly quality of each component belonging to the detection image during visual inspection.

[0052] The term "corrected image" may refer to an image that is converted to be close to (similar to) the reference image by compensating for the deviation of the detection image relative to the reference image.

[0053] The detected component image may refer to a single component image extracted from the corrected image corresponding to the detected component region ROI set in the reference image.

[0054] The learning image may refer to a deep learning learning dataset enhanced by image enhancement by converting the detected component image.

[0055] Now, a vision inspection system and method using a mobile robot according to an embodiment will be described in detail with reference to the accompanying drawings.

[0056] Figure 1 A vision inspection system using a mobile robot applied to a product assembly process according to an embodiment is shown.

[0057] Figure 2 It is a block diagram schematically showing the configuration of a vision inspection system using a mobile robot according to an embodiment.

[0058] Referring to Figure 1 and Figure 2 , the detection server 100 according to an embodiment evaluates the component assembly quality of workers in the product assembly process of an industrial site, may include a mobile robot 10 configured to move to at least one specified detection position P and capture the detection area of the components assembled with the product, and may be configured to obtain the detection image captured by the mobile robot 10, automatically calculate the image deviation of the detection image compared with the reference image, and evaluate the assembly quality of the detection image at each detection position through a detection algorithm, the image deviation reflecting the repetitive position error of the robot for each detection position, and the detection algorithm being learned by performing image enhancement reflecting the range of the image deviation.

[0059] Here, the product is transported to a predetermined working position by the transport device 20 of the intelligent factory and at least one component assigned to the worker is assembled. Hereinafter, it is assumed that the product is a "vehicle", and the vehicle can be a body or a component of the body (for example, a door, an instrument panel, an interior trim, etc.) in the assembly process. The transport device 20 can be a conveyor belt, a logistics transport robot, etc.

[0060] The mobile robot 10 can move to the designated detection position P and send the captured detection image to the detection server 100. Here, the detection position P can be a shooting position / location existing in a plurality of working areas (for example, the first area to the sixth area) divided around the vehicle, and for example, can include the first detection position P1, the second detection position P2,..., the sixth detection position P6 respectively designated for the plurality of working areas. However, the number of working areas and detection positions in the embodiment is not limited thereto, and according to the assembly position of the component, a plurality of detection positions can be designated in the same working area.

[0061] The mobile robot 10 can be configured as a quadruped walking robot, an autonomous mobile robot (AMR), etc. The AMR is limited to moving on a flat surface by using drive wheels. The preferred embodiment of the present disclosure will assume that a quadruped walking robot (also called a "machine dog" or "spot") can move on stairs or uneven rough surfaces with a higher degree of freedom for description.

[0062] As Figure 2 shown, the mobile robot 10 can include a visual sensor module 11, an autonomous driving sensor module 12, a mobile module 13, a wireless communication module 14, and a control module 15.

[0063] The visual sensor module 11 can be installed on the robot and can generate a detection image captured at the designated detection position P.

[0064] In order to change the degree of freedom of the shooting position, the visual sensor module 11 can be installed on the mobile robot 10 through the robot arm 11-1. By utilizing the attitude control of the robot arm 11-1, the visual sensor module 11 can not only shoot the external components of the vehicle, but also shoot the internal components inside the vehicle.

[0065] The autonomous driving sensor module 12 can include at least one sensor among a camera, a laser, an ultrasonic wave, a radar, a lidar, and an autonomous driving position recognition device to detect the surrounding environment and identify workers and objects.

[0066] The mobile module 13 can include four legs and can move freely on stairs or uneven surfaces by quadruped walking.

[0067] The wireless communication module 14 can transmit the detection image captured by the vision sensor module 11 to the detection server 100 via wireless communication, and can receive control signals from the detection server 100 when needed.

[0068] According to an embodiment, the control module 15 can control the overall operation of the mobile robot 10.

[0069] When the vehicle is transported to the working position, the worker can assemble at least one component assigned to the worker to the vehicle. The components can be the body, parts, electrical components, etc. to be assembled to the vehicle, or can include fastening components such as bolts, nuts, rivets, etc. with specified assembly positions.

[0070] The control module 15 can store the assembly positions of the components to be assembled in the corresponding processes according to the type and specifications of the vehicle, as well as at least one detection position P1, P2,..., P6 specified for capturing the detection image.

[0071] The control module 15 can perform control to identify the position of the worker through the vision sensor module 11, follow the worker to move to the detection positions P1, P2,..., P6 where the work has been completed, and capture the detection image through the vision sensor module 11.

[0072] For example, as Figure 1 shown, when the worker works in the fourth area after completing work in the first, second, and third areas, the mobile robot 10 can move in the order of P1, P2, and P3, and can wait after capturing the detection image.

[0073] At this time, the control module 15 can restrict the movement of the mobile robot 10 so as not to enter the worker area (i.e., the fourth area) where the worker is currently located and wait, thereby preventing potential collisions without interfering with the worker's work path.

[0074] In addition, for the safety of the worker, a monitoring camera 30 can be further configured in the assembly process area. The monitoring camera 30 is configured to monitor the position of the worker and send an event that the mobile robot 10 enters the worker area to the detection server 100.

[0075] The detection server 100 can immediately send a stop signal to the mobile robot 10 when receiving the entry event to restrict the movement to the worker area, thereby ensuring the safety of the worker.

[0076] On the other hand, when obtaining the detection image from the mobile robot 10, the detection server 100 can detect feature points to perform image conversion to approach the reference image, and detect at least one detected component image extracted by cropping the detected component area (region of interest, ROI) through a deep learning vision detection program.

[0077] However, when performing visual inspection by using the mobile robot 10, there is a problem that in the inspection image obtained based on the position of the mobile robot 10 and the position of the inspected component, due to the position error occurring during shooting, an image deviation occurs with respect to the reference image.

[0078] For example, Figure 3 is a diagram for explaining the problem of image deviation occurring during visual inspection using a mobile robot according to an embodiment.

[0079] Referring to Figure 3 , the reference image 121 is the best image obtained by shooting the inspection area after previously positioning the mobile robot 10 at the first inspection position P1.

[0080] The reference image 121 can be a reference for visual inspection for evaluating the assembly quality of the components belonging to the corresponding inspection image, and at least one inspected component ROI area and the entire ROI area including it can be set.

[0081] However, since the mobile robot 10 is a mobile device, different from existing fixed devices, even if it is set to move to the specified inspection positions P1, P2,..., P6 to shoot the inspection area of the vehicle body, the inspection images taken each time deviate due to minute repetitive position errors (hereinafter referred to as "repetitive position errors"), resulting in a decline in inspection performance.

[0082] That is, as Figure 3 shown, each time the mobile robot 10 moves to the first inspection position P1 and obtains an inspection image, repetitive position errors of large or small (e.g., 10 mm to 500 mm) may occur.

[0083] Therefore, when the mobile robot 10 performs multiple inspection shootings at the actual position, image deviations from the reference image 121 may occur in the first inspection image, the second inspection image, the third inspection image, etc. Here, when the mobile robot 10 is used by being attached to the robot arm 11-1 for the freedom of image shooting, the deviation of the taken inspection image may further increase according to the repetitive position error and the position error of the robot arm 11-1.

[0084] Therefore, in the inspection image obtained from the mobile robot 10, an image deviation may occur due to the repetitive position error of the robot, which can be understood as the translation, rotation, scaling, inclination, and shear of the image generated by the repetitive position error being reflected in the image deviation.

[0085] Considering these problems, when performing visual inspection by using the mobile robot 10, when compensating for the image deviation of the obtained inspection image, the image conversion conditions may repeatedly change due to the repeated position error of the robot, and the inspection server 100 must generate an inspection algorithm that reflects the repeated position error of the components for each inspection position P of the robot.

[0086] Therefore, the inspection server 100 according to the embodiment attempts to automatically generate an inspection algorithm that reflects the image deviation range (repeated position error) of the components for each inspection position P of the mobile robot 10 through deep learning.

[0087] For this purpose, the inspection server 100 can generate an inspection algorithm by performing image enhancement that reflects the repeated position error range of the mobile robot 10 when performing deep learning, and can improve the inspection performance by performing visual inspection with an inspection algorithm that considers the repeated position error of the mobile robot 10.

[0088] The inspection server 100 may include a communication unit 110, an image processing unit 120, a deep learning unit 130, a database (DB) 140, and a controller 150.

[0089] The communication unit 110 may include a wired / wireless communication device and may obtain an inspection image from the mobile robot 10.

[0090] The image processing unit 120 may store a reference image corresponding to the inspection image of each component and may generate a corrected image obtained by performing a matching process on the obtained inspection image by comparing it with the corresponding reference image.

[0091] Figure 4 Represents an inspection image processing method for visual inspection according to an embodiment.

[0092] Figure 5 Represents a transformation matrix according to an embodiment.

[0093] Refer to Figure 4 and Figure 5 and

[0094] When performing image matching processing, the image processing unit 120 may calculate the image matching error of the inspection image with respect to the reference image and calculate the image deviation between the two images to extract the transformation matrix information that can be obtained when correcting the inspection image. Here, the transformation matrix information may include information on translation for image movement, rotation for image rotation, scale for image magnification / shrinking, tilt for image slope conversion, and shear for image shearing, as values for quantifying the image deviation.The image processing unit 120 may extract at least one detected component image corresponding to the detected component region of interest (ROI) set for the reference image in the corrected image as learning data.

[0095] On the other hand, Figure 6 represents a method for constructing a detection algorithm for each detected component according to an embodiment.

[0096] Figure 7 represents a visual detection method using the detection algorithm of each detected component according to an embodiment.

[0097] Referring to Figure 6 and Figure 7 , the deep learning unit 130 may construct a detection algorithm for each detected component P1-1, P1-2, P2,..., P6 at each detection position P1, P2,..., P6 of the mobile robot 10 by pre-enhancing the detection image and learning the repeated position error range.

[0098] At this time, the deep learning unit 130 may perform image enhancement using the transformation matrix range and the detected component image extracted by the image processing unit 120, thereby enhancing the learning image reflecting the repeated position error range of the corresponding detected component. According to the transformation matrix range, a range of random numbers for the transformation values of translation, rotation, scaling, skew, and shear of the image can be specified for the learning image. Here, image enhancement can be performed on the multiple detected component images included in one detection image by equally applying the transformation matrix range.

[0099] In addition, the deep learning unit 130 may enhance the randomly scaled image reflecting the repeated position error range of the robot and apply it to the learning image.

[0100] In this way, by enhancing various learning images reflecting the repeated position error range of the robot for the specified detection position to generate a detection algorithm, the mobile robot can expand the detectable area around the detection position. Therefore, even if the repeated position error of the mobile robot appears within the detectable area, detection errors can be prevented and detection performance can be improved.

[0101] The deep learning unit 130 may learn the detected component image through deep learning by the detection algorithm of the corresponding component and output one of the detection results of good (OK), defective (NG), and detection error (NA).

[0102] For example, in response to the input of the P1-1 detected component image, the deep learning unit 130 may perform visual detection through the corresponding P1-1 algorithm and output the detection result.

[0103] Here, the inspection result can be determined based on the similarity ratio (%) of the component image detected by P1-1 to the good product reference image P1-1(OK) of the corresponding component assembled normally or to the defective reference image P1-1(NG) of the unassembled component. However, when the component image detected by P1-1 is not close to either the good product reference image P1-1(OK) or the defective reference image P1-1(NG), an inspection error (NA) can be output, which means that it cannot be determined.

[0104] In addition, the deep learning unit 130 can determine whether it is good / defective (OK / NG) based on the condition that the first detected component image P1-1 and the good product reference image (P1-1(OK)) match with a predetermined ratio (e.g., 80% or more or less).

[0105] The deep learning unit 130 can be configured based on a program of an artificial neural network.

[0106] The DB 140 can store various programs and data required for the operation of the inspection server 100 using the mobile robot according to the embodiment, and can store the data generated according to its operation.

[0107] The DB 140 can store the inspection algorithms for each detected component P1-1, P1-2, P2, ……, P6 corresponding to the respective processes, and can replace or additionally update the newly generated inspection algorithms in the case of repeated visual inspections.

[0108] The controller 150 can be a central processing unit configured to control the overall operation of the inspection server 100 that performs visual inspection using the mobile robot according to the embodiment. That is, the controller 150 can control each component configured in the server 100 by executing various programs stored in the DB 140.

[0109] The controller 150 can be implemented using one or more processors operating according to a set program, and the set program can be programmed to execute each step of the visual inspection method according to the embodiment.

[0110] The visual inspection method using the mobile robot will be described in more detail below with reference to the drawings.

[0111] Figure 8 is a flowchart schematically showing the visual inspection method using the mobile robot according to the embodiment.

[0112] Refer to Figure 8 as described above with reference to Figure 1As described above, a vision detection method using a mobile robot according to an embodiment will be described in the following scenario: In order to perform vision detection on the manual assembly process of components, the moving robot 10 in operation moves to the specified first detection position P1 to the sixth detection position P6 around the product vehicle, and captures detection images of the components assembled with the vehicle.

[0113] In step S110, the controller 150 of the detection server 100 may obtain a detection image captured at a specific detection position that needs to be detected around the product vehicle by using the moving robot 10 in operation. Hereinafter, for better understanding and convenience of description, it is assumed that the controller 150 obtains a detection image captured at the first detection position P1 from the moving robot 10.

[0114] The controller 150 may generate a corrected image, which is converted by performing image matching processing on the detection image obtained from the moving robot 10 to match the detected component region of interest (ROI) of a preset reference image. Additionally, in step S120, at least one detected component image corresponding to the detected component region of interest (ROI) of the reference image is extracted from the corrected image. For example, the detected component image has a component number P1-1 corresponding to the first detection position P1, and through this, the detected component P1-1 that is the object of vision detection can be identified.

[0115] The controller 150 automatically calculates in advance the image deviation reflecting the repeated position error of the robot for each detection position when comparing the detection image with the reference image, and constructs a detection algorithm for the assembled components P1-1, P1-2, P2, ……, P6 of the corresponding process learned through deep learning by performing automatic image enhancement reflecting the range of the image deviation.

[0116] In step S130, the controller 150 performs vision detection by using deep learning of the detection algorithm of the first detected component P1-1 corresponding to the first detected component image P1-1.

[0117] In step S140, the controller 150 may obtain one of the detection results of good (OK), defective (NG), and detection error (NA) according to the vision detection.

[0118] At this time, when the detection result is determined to be good (OK), the controller 150 may terminate the corresponding detection and return to repeat the vision detection for the subsequent component detection images P1-2, P2, ……, P6, and when there are no subsequent component detection images, the detection may be terminated.

[0119] On the other hand, when the detection result is determined to be a defect (NG) or a detection error (NA), the controller 150 can extract the images that are misdetected (e.g., misjudgment between OK and NG) or cannot be determined through the operator's confirmation of the detection result, store these images, and use these images as evaluation data for confirming the performance of the new detection algorithm described later in step S160.

[0120] In this way, the controller 150 can pre-learn the range of repeat errors at each detection position of the mobile robot 10 to construct the detection algorithms for the components P1-1, P1-2, P2, ……, P6, and perform visual detection using this, thereby compensating for the image deviation due to the repeat position error of the mobile robot 10, and thus improving the evaluation performance of the assembly quality of each component.

[0121] On the other hand, in the pre-constructed detection algorithm (hereinafter referred to as the “existing detection algorithm”), the image deviation may be caused by various environmental changes in the process operation such as the aging of the process equipment including the mobile robot 10 or the change in the installation position of the vision sensor module 11 over time, and due to this, the detection performance may decrease.

[0122] Therefore, in order to prevent the detection performance from decreasing, the controller 150 is characterized in that it generates a new detection algorithm based on the transformation matrix information obtained during the image matching process in step S120, and the image enhancement range of real-time deep learning is optimized in the new detection algorithm.

[0123] Thus, by replacing or additionally updating the existing detection algorithm with the new detection algorithm that reflects various environmental changes, it is possible to prevent the performance degradation of the detection algorithm not only due to the repeat position error of the mobile robot but also due to various environmental changes such as equipment aging, and in addition, the detection performance can be further improved.

[0124] Hereinafter, a method for generating a new detection algorithm for each component according to the embodiment will be described in detail.

[0125] In step S121, during the image matching process, the controller 150 calculates the image matching error of the detection image relative to the reference image. In addition, in step S122, the image deviation between the two images is calculated to extract the transformation matrix that can be obtained when correcting the detection image. Here, the transformation matrix can include information on translation for image movement, rotation for image rotation, scaling for image enlargement / reduction, skew for image slope conversion, and shear for image shearing as values for quantifying the image deviation.

[0126] In step S123, the controller 150 calculates the distribution of the transformation matrix values corresponding to the detected (captured) positions of the mobile robot 10, and forms a DB of this range to store it in the DB 140.

[0127] In step S124, the controller 150 designates the range of the transformation matrix for image enhancement during the deep learning of the corresponding detected component images based on the range of repeated position errors stored in the DB 140. That is, the controller 150 can designate the range of random number generation of the conversion values of translation, rotation, scaling, inclination, and shear of the detected component images by using enhancement during deep learning based on the range of repeated position errors of the transformation matrix. In particular, the controller 150 can generate random numbers according to the Gaussian normal distribution when designating the transformation matrix range, and can perform deep learning by assigning a greater weight value to the image deviation with a high possibility of generating repeated errors of the mobile robot according to the position.

[0128] In step S125, the controller 150 can generate a new detection algorithm for the corresponding component by deep learning the multiple learning images enhanced by reflecting the repeated position errors of the mobile robot 10, and can re-learn the detected component images by deep learning using the new detection algorithm.

[0129] That is, when performing deep learning, the controller 150 can generate a new detection algorithm for each component by image enhancement reflecting the error range for each position of the mobile robot 10, and by re-learning through deep learning using this, the detection performance can be improved.

[0130] In step S126, the controller 150 compares the evaluation data of the existing detection algorithm provided for the performance evaluation of the new detection algorithm with the re-learned detection results.

[0131] At this time, when the evaluation performance of the new detection algorithm is improved compared to the pre-built existing detection algorithm (for example, existing P1-1 algorithm < new P1-1 algorithm) (S126 - Yes), the controller 150 automatically replaces the existing detection algorithm with the new algorithm. For example, in the case where an image with a detection error (NA) by using the existing detection algorithm can be determined as good (OK) or defective (NG) by using the new detection algorithm, it is determined that the evaluation performance is improved, and the algorithm can be automatically replaced.

[0132] On the other hand, when the evaluation performance is not improved and is lower than or equal to the existing detection algorithm (S126 - No), the controller 150 can maintain the existing detection algorithm.

[0133] Thus, according to the embodiment, when performing visual inspection using a mobile robot, an image deviation reflecting the repetitive position error of the mobile robot is automatically calculated, and visual inspection is performed by generating a detection algorithm reflecting the image deviation, thereby improving the detection performance for the repetitive position error of the mobile robot.

[0134] In addition, when performing visual inspection, a new detection algorithm is generated by specifying an enhancement range reflecting the image deviation due to device aging over time or various environmental changes, and the existing detection algorithm is replaced / updated with the new detection algorithm, which can prevent the degradation of detection performance and maintain the optimal detection quality.

[0135] In addition, the mobile robot can follow a worker to perform visual inspection and restrict its movement so that it does not enter the area where the worker is currently located, thereby preventing collisions without interfering with the worker's work path.

[0136] The above-described exemplary embodiments of the present disclosure can be implemented not only by devices and methods but also by a program for implementing functions corresponding to the configurations of the embodiments of the present disclosure or a recording medium recording the program.

[0137] Although the present disclosure has been described in connection with currently considered practical embodiments, it will be understood that the present disclosure is not limited to the disclosed embodiments. On the contrary, the present disclosure aims to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A vision detection system using a mobile robot, comprising: A mobile robot configured to move to at least one specified detection position (P) and capture a detection area of components assembled with a product; and A detection server configured to obtain a detection image captured by the mobile robot, calculate an image deviation of the detection image compared with a reference image, and evaluate the assembly quality of the detection image at each detection position through a detection algorithm, where the image deviation reflects the repeated position error of the mobile robot for each detection position, and the detection algorithm is learned by performing image enhancement reflecting the range of the image deviation.

2. The vision detection system according to claim 1, wherein, The mobile robot includes: A vision sensor module configured to generate the detection image captured at the detection position; An autonomous driving sensor module configured to detect the surrounding environment through sensors; A mobile module configured to move freely by driving wheels or quadruped walking; A wireless communication module configured to send the captured detection image to the detection server through wireless communication; and A control module configured to perform control to identify the position of a worker through the vision sensor module, follow the worker to move to the detection position (P) where the work has been completed, and capture the detection image through the vision sensor module.

3. The vision detection system according to claim 2, wherein, The control module stores the assembly positions of components to be assembled in corresponding processes according to the type and specifications of the product, and at least one detection position designated for capturing the detection image of the components.

4. The vision detection system according to claim 2, wherein, The control module is configured to distinguish multiple regions divided around the product and restrict movement to not enter the worker area where the worker is currently located.

5. The vision detection system according to claim 2, wherein, The vision sensor module is mounted on the mobile robot through a robotic arm and is configured to capture detection images of external and internal components of the product through attitude control of the robotic arm.

6. The vision detection system according to claim 1, wherein, The detection server includes: A communication unit configured to obtain a detection image from the mobile robot; An image processing unit configured to store a reference image corresponding to the detection image of each component and generate a corrected image converted by performing matching processing on the detection image through comparison with the reference image; A deep learning unit configured to construct a detection algorithm for each detected component in the corresponding process by pre-enhancing the detection image and learning the range of repeated position errors of each detection position of the mobile robot through deep learning; A database (DB) configured to store programs and data for operating the detection server; and A controller configured to generate a new detection algorithm by specifying an image enhancement range for deep learning based on the transformation matrix obtained during the image matching process.

7. The visual inspection system according to claim 6, wherein, when performing image matching processing, the image processing unit calculates the image matching error of the detection image relative to the reference image, and calculates the image deviation between the two images, so as to extract the transformation matrix information that can be obtained when correcting the detection image.

8. The visual inspection system according to claim 7, wherein, the transformation matrix information includes information about translation for image movement, rotation for image rotation, scaling for image enlargement / reduction, skew for image slope conversion, and shear for image shearing, as values for quantifying the image deviation.

9. The visual inspection system according to claim 6, wherein, the image processing unit is configured to extract at least one detected component image corresponding to the detected component region (ROI) set for the reference image in the corrected image as learning data.

10. The visual inspection system according to any one of claims 6 to 9, wherein, the deep learning unit is configured to perform image enhancement by using the transformation matrix range and the detected component images extracted by the image processing unit, and enhance the learning images reflecting the range of the repeated position error of the corresponding detected components.

11. The visual inspection system according to claim 10, wherein, the deep learning unit is configured to learn the detected component images through deep learning by means of the detection algorithms of the corresponding components, and output one of the detection results of good (OK), defective (NG), and detection error (NA).

12. The visual inspection system according to claim 11, wherein, the controller is configured to generate a new detection algorithm that reflects the image enhancement range of the deep learning in real time based on the transformation matrix information obtained during the image matching processing of the detection image.

13. The visual inspection system according to claim 11, wherein, the controller is configured to, when it is determined that the performance has been improved by comparing the detection results obtained by re-learning through deep learning using the new detection algorithm with the detection results obtained by using the existing detection algorithm, replace or update the existing detection algorithm with the new detection algorithm.

14. A visual inspection method using a mobile robot for real-time inspection of the manual assembly quality of workers in a vehicle assembly process, the visual inspection method comprises: obtaining a detection image taken at a specific detection position around the product vehicle that needs to be detected by using the mobile robot; extracting a detected component image from the corrected image, the corrected image being converted by performing image matching processing on the detection image to match a preset reference image; automatically calculating the image deviation reflecting the repeated position error of the robot for each detection position of the detection image compared with the reference image, storing the detection algorithm learned by performing image enhancement reflecting the range of the image deviation, and performing visual inspection through deep learning of the detection algorithm corresponding to the detected component of the detected component image; and Obtain one of the inspection results of Good (OK), Defect (NG), and Inspection Error (NA) based on the visual inspection.

15. The visual inspection method using a mobile robot according to claim 14, wherein, obtaining the inspection result further includes: When the inspection result is determined to be Defect (NG) or Inspection Error (NA), extract the images that are misdetected or cannot be determined through the operator's confirmation of the inspection result, and store the extracted images as evaluation data for confirming the performance of the new detection algorithm.

16. The visual inspection method using a mobile robot according to claim 14 or 15, wherein, extracting the inspection component images further includes: Generate a new detection algorithm by specifying the image enhancement range learned by the deep learning based on the transformation matrix obtained during the image matching process.

17. The visual inspection method using a mobile robot according to claim 16, wherein, generating the new detection algorithm includes: Calculate the image matching error of the inspection image relative to the reference image; Extract the transformation matrix by calculating the image deviation between two images, which can be obtained when correcting the inspection image; Calculate the distribution of the transformation matrix values corresponding to the inspection positions and store the range in the DB; Based on the repeated position error range stored in the DB, specify the transformation matrix range for image enhancement during the deep learning of the corresponding inspection component images; and Deep learning learns multiple learning images enhanced by reflecting the repeated position error of the mobile robot, generates a new detection algorithm for the corresponding components, and relearns the inspection component images through deep learning.

18. The visual inspection method using a mobile robot according to claim 17, wherein, Specifying the matrix range uses the enhancement during deep learning based on the repeated position error range of the transformation matrix to specify the range of random numbers generated for the conversion values of translation, rotation, scaling, skew, and shear of the inspection component images.

19. The visual inspection method using a mobile robot according to claim 18, wherein, Specifying the matrix range generates random numbers according to the Gaussian normal distribution, and performs the deep learning by assigning weight values to the image deviations with a high probability of generating the repeated error of the mobile robot according to the position.

20. The visual inspection method using a mobile robot according to claim 17, further includes: After relearning through deep learning, when it is determined that the performance has improved by comparing the relearning result with the evaluation data, use the new detection algorithm to replace or update the existing detection algorithm.

Images