3D reconstruction system and method thereof

Abstract

The present invention relates to 3D reconstruction technology. A 3D reconstruction system according to an embodiment of the present invention may comprise: a position checking device for checking the position of a vehicle entering a work space; a scanning device for scanning the interior and exterior of the vehicle entering the work space, while scanning vehicle components located within the work space; a 3D depth information generation device for generating 3D depth information about the vehicle and the vehicle components on the basis of the scan images generated by the scanning device; and a controller for controlling the scanning device on the basis of the information generated by the position checking device and the information generated by the scanning device.

Description

3D reconstruction system and method thereof

[0001] The present invention relates to 3D reconstruction technology, and more particularly, to a 3D reconstruction system and method applicable to an automatic vehicle assembly line requiring a robot and a scanning device.

[0002] In computer vision, 3D reconstruction is the process of determining the shape and / or appearance of real-world objects. 3D reconstruction can be based on data and / or images of the object obtained from various types of sensors.

[0003] 3D reconstruction technologies include lidar-based systems, structured light line-based systems, and stereo vision-based systems.

[0004] Lidar-based systems utilize the time-of-flight (TOF) principle by projecting pulses of light into the surrounding environment that reflect off objects and return to the sensor's detector. However, they only provide sparse depth, which does not provide sufficient data for measurement, and they are bulky.

[0005] Structured light line-based systems project a structured light pattern onto the surface of a measurement target and then simultaneously track the light pattern using a camera to calculate the object's position. However, they have the problem that not only does it take a relatively long time to acquire data, but it is also difficult to apply to large objects at a long distance.

[0006] Stereo vision-based systems obtain 3D depth information from images captured by two cameras positioned parallel to the same baseline, but have the problem of being sensitive to textureless and reflective objects commonly found in manufacturing plants.

[0007] Conventional 3D reconstruction technology has limitations in that it is not suitable for application to automatic vehicle assembly lines that require robots and scanning devices.

[0008] Therefore, a 3D reconstruction technology is required that can acquire 3D depth information more efficiently than conventional technologies and can be applied to vehicle automatic assembly lines that require robots and scanning devices.

[0009] This background technology is technical information that the inventor possessed for the purpose of deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be considered as publicly known technology disclosed to the general public prior to the application for the present invention.

[0010] The embodiments disclosed in the present invention are conceived in accordance with the aforementioned needs, and the technical task is to provide a 3D reconstruction system and method thereof that can be applied to an automatic vehicle assembly line requiring a robot and a scanning device.

[0011] In addition, the embodiment is a technical problem to provide a 3D reconstruction system and method capable of performing both scanning of a vehicle and scanning of a vehicle part in one workspace.

[0012] In addition, the embodiment is a technical problem of providing a 3D reconstruction system and method capable of acquiring in real time, while scanning, movement information related to a robot arm / end-effector used to estimate the posture of a scan sensor of a scanning device.

[0013] In addition, the embodiment is a technical problem of providing a 3D reconstruction system and method capable of obtaining position change information for a mounting point of a vehicle / vehicle component.

[0014] The technical problems of the present invention are not limited to those mentioned above, and those with ordinary skill in the art to which the present invention pertains will be able to clearly understand other problems intended by the present invention from the following description.

[0015] As a technical means for achieving the above-described technical task, a 3D reconstruction system and method thereof that can be applied to an automatic vehicle assembly line requiring a robot and a scanning device can be provided.

[0016] A 3D reconstruction system according to an embodiment of the present invention may include a position confirmation device for confirming a position of a vehicle entering a work space, a scanning device for scanning the interior and exterior of the vehicle entering the work space while scanning vehicle parts located within the work space, a 3D depth information generation device for generating 3D depth information on the vehicle and the vehicle parts based on a scan image generated by the scanning device, and a controller for controlling the scanning device based on the information generated by the position confirmation device and the information generated by the scanning device.

[0017] According to an embodiment, the positioning device may include a tag installed in a vehicle and providing its own position information, and a position information generation unit that generates position and attitude information about the vehicle based on the position information provided from the tag.

[0018] According to an embodiment, the controller can compare 3D depth information generated by the 3D depth information generating device with an original CAD model and calculate a change in the position of a mounting point of a vehicle and vehicle parts.

[0019] According to an embodiment, the scanning device may include a first scanning device that scans the exterior of a vehicle located in a first scan area set outside an entrance of a work space, a second scanning device that scans the exterior of a vehicle located in a second scan area set inside the work space and adjacent to the entrance of the work space, and a third scanning device that scans the interior of a vehicle located in a third scan area set inside the work space while scanning a vehicle part located in a fourth scan area inside the work space.

[0020] According to an embodiment, the first and second scanning devices may include robot arms equipped with scan sensors, and the third scanning device may include a multi-joint robot arm including a scan sensor.

[0021] According to an embodiment, the scanning device includes a plurality of first scanning devices, and the plurality of first scanning devices are capable of scanning the exterior of the vehicle within the first scan area at different locations.

[0022] According to an embodiment, the scanning device includes a plurality of second scanning devices, and the plurality of second scanning devices are capable of scanning the exterior of the vehicle within the second scan area at different locations.

[0023] According to an embodiment, the first to third scanning devices may perform a scanning operation alone or together, depending on the location of the vehicle.

[0024] According to an embodiment, a conveyor may be installed within a work space, and a third scanning device may be installed on the conveyor.

[0025] According to an embodiment, the third scanning device can generate a scan trajectory based on vehicle-related position and attitude information generated by the positioning device, and scan the interior of the vehicle while moving along the scan trajectory.

[0026] According to an embodiment, the third scanning device can scan the interior of the vehicle while moving according to scan trajectory information provided from the controller.

[0027] According to an embodiment, the controller may control a third scanning device to scan a vehicle component if the controller determines that the vehicle has left a preset scan area based on location information about the vehicle generated by the positioning device.

[0028] In an embodiment, the third scanning device may generate movement information for the robot arm and end-effector while scanning the interior of the vehicle.

[0029] In an embodiment, the third scanning device may generate motion information for the robot arm and end-effector while scanning the vehicle part.

[0030] In an embodiment, the controller may control the third scanning device to scan a vehicle component if the controller determines that the robot arm is in an idle state based on movement information generated by the third scanning device while scanning the interior of the vehicle.

[0031] A 3D reconstruction method according to an embodiment of the present invention may include a step of confirming the position of a vehicle entering a work space using a position confirmation device, a step of scanning the interior and exterior of the vehicle entering the work space using a scanning device, a step of scanning a vehicle part located within the work space using the scanning device, and a step of generating 3D depth information for the vehicle and the vehicle part using a 3D depth information generating device based on a scan image generated by the scanning device.

[0032] According to an embodiment, the step of scanning the interior and exterior of the vehicle using a scanning device may include the step of scanning the exterior of the vehicle located in a first scan area set outside an entrance of the work space using a first scanning device, the step of scanning the exterior of the vehicle located in a second scan area set inside the work space and adjacent to the entrance of the work space using a second scanning device, and the step of scanning the interior of the vehicle located in a third scan area set inside the work space using a third scanning device.

[0033] In an embodiment, the step of scanning using the third scanning device may include generating motion information for a multi-joint robot arm and an end-effector included in the third scanning device while scanning the interior of the vehicle.

[0034] According to an embodiment, the step of scanning a vehicle part using the scanning device may include scanning a vehicle part located in a fourth scan area set within a work space using a third scanning device.

[0035] In an embodiment, the step of scanning a vehicle part using the scanning device may include generating movement information for a multi-joint robot arm and an end-effector included in a third scanning device while scanning the vehicle part.

[0036] According to an embodiment, the step of identifying using a location identification device may include a tag installed in a vehicle providing its own location information, and a location information generation unit generating location and attitude information for the vehicle based on the location information from the tag.

[0037] According to an embodiment, the 3D reconstruction method may further include a step in which the controller compares 3D depth information generated by the 3D depth information generating device with an original CAD model and calculates a change in position of mounting points of the vehicle and vehicle components.

[0038] According to an embodiment, the step of scanning using the first scanning device may be performed using a plurality of first scanning devices that scan the exterior of the vehicle within the first scan area at different locations.

[0039] In an embodiment, the step of scanning using a second scanning device may be performed using a plurality of second scanning devices that scan the exterior of the vehicle within the second scan area at different locations.

[0040] According to an embodiment, the step of scanning the interior and exterior of the vehicle using a scanning device may include, depending on the location of the vehicle, the first to third scanning devices performing a scanning operation alone.

[0041] According to an embodiment, the step of scanning the interior and exterior of the vehicle using a scanning device may include, depending on the location of the vehicle, the first to third scanning devices performing a scanning operation together.

[0042] According to an embodiment, the step of scanning the interior of the vehicle using the third scanning device may include the third scanning device generating a scan trajectory based on vehicle-related position and attitude information generated by the positioning device, and scanning the interior of the vehicle while moving along the scan trajectory.

[0043] Specific details according to various examples of the present invention other than the means for solving the problems mentioned above are included in the description and drawings below.

[0044] According to the present invention, by performing path planning for a logistics robot based on a directional movement area, a deadlock situation of a logistics robot can be prevented. According to an embodiment of the present invention, a 3D reconstruction system and method thereof that can be applied to an automatic vehicle assembly line requiring a robot and a scanning device can be provided.

[0045] In an embodiment, since both scanning of vehicles and scanning of vehicle parts are possible in one workspace, the time required to scan vehicles and vehicle parts can be shortened.

[0046] According to an embodiment, since movement information related to a robot arm / end-effector used to estimate the posture of a scan sensor of a scanning device can be acquired in real time during scanning, the time required to estimate the posture of a scan sensor of a scanning device can be shortened.

[0047] According to an embodiment, since position change information for mounting points of vehicles / vehicle components can be obtained, a tolerance model for manufacturing errors in components and processes can be constructed.

[0048] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0049] The contents of the problem to be solved, the means for solving the problem, and the effect mentioned above do not specify the essential features of the claims, so the scope of the claims is not limited by the matters described in the contents of the invention.

[0050] The accompanying drawings are intended to aid in understanding embodiments of the present invention and provide examples along with detailed descriptions. However, the technical features of the embodiments are not limited to specific drawings, and the features disclosed in each drawing may be combined to form new embodiments.

[0051] FIG. 1 is a block diagram showing the configuration of a 3D reconstruction system according to an embodiment of the present invention.

[0052] Figure 2 is a schematic drawing of a work space according to an embodiment of the present invention.

[0053] FIG. 3 is a drawing showing an example of a scanning device according to an embodiment of the present invention scanning the exterior of a vehicle located in a first scan area.

[0054] FIG. 4 is a drawing showing an example of a scanning device according to an embodiment of the present invention scanning the exterior and interior of a vehicle positioned across first to third scan areas.

[0055] FIG. 5 is a drawing showing an example of a scanning device according to an embodiment of the present invention scanning a vehicle component located in a fourth scan area.

[0056] FIG. 6A is a front view of a scan sensor included in a scan device according to an embodiment of the present invention.

[0057] Fig. 6b is a schematic drawing of the scan sensor viewed from direction A of Fig. 6a.

[0058] Fig. 6c is a schematic drawing of the scan sensor as seen from the B direction of Fig. 6a.

[0059] FIG. 7 is a drawing for explaining a 3D reconstruction method according to an embodiment of the present invention.

[0060] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined solely by the scope of the claims.

[0061] The shapes, sizes, ratios, angles, numbers, etc. disclosed in the drawings for explaining embodiments of the present invention are illustrative and are not limited to the matters illustrated. Like reference numerals refer to like components throughout the specification. In addition, in describing the present invention, if a detailed description of a related known technology is judged to unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. When "comprises," "has," "consists of," etc. are used in this specification, other parts may be added unless "only" is used. When a component is expressed in the singular, it includes a case where the plural is included unless specifically stated otherwise.

[0062] When interpreting a component, it is interpreted as including the error range even if there is no separate explicit description of the error range.

[0063] When describing a temporal relationship, if the temporal continuity is described as "after", "continuously", "next to", "before", etc., it can also include cases where it is not continuous, unless "immediately" or "directly" is used.

[0064] While terms like "first" and "second" are used to describe various components, these components are not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, a "first" component referred to below may also be a "second" component within the technical scope of the present invention.

[0065] In describing components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only intended to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by the terms. When a component is described as being "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that other components may also be "interposed" between each component that is indirectly connected or connected without any specifically explicit description.

[0066] "At least one" should be understood to include any combination of one or more of the associated components. For example, "at least one of the first, second, and third components" could be understood to include not only the first, second, or third components, but also any combination of two or more of the first, second, and third components.

[0067] The individual features of the various embodiments of this specification may be partially or wholly combined or combined with each other, and may be technically linked and operated in various ways, and each embodiment may be implemented independently of each other or may be implemented together in a related relationship.

[0068] Hereinafter, embodiments of the present invention will be described with reference to the attached drawings and examples. The scale of the components illustrated in the drawings is different from the actual scale for convenience of explanation, and is therefore not limited to the scale illustrated in the drawings.

[0069] Hereinafter, a method and system for improving the picture quality of a vehicle display device according to an embodiment of the present invention will be described with reference to the attached drawings.

[0070]

[0071] FIG. 1 is a block diagram showing the configuration of a 3D reconstruction system (1) according to an embodiment of the present invention, and FIG. 2 is a drawing schematically showing a work space according to an embodiment of the present invention.

[0072] For the purpose of explanation, the following description will be made using an example in which a 3D reconstruction system (1) according to an embodiment of the present invention performs a task targeting a vehicle, but the 3D reconstruction system (1) according to an embodiment of the present invention is not only applicable to a vehicle.

[0073] A 3D reconstruction system (1) according to an embodiment can scan a vehicle and vehicle parts (or vehicle parts) of a work target (here, a scan target), and perform 3D reconstruction based on the scan results to generate 3D depth information about the vehicle.

[0074] The 3D reconstruction system (1) can scan the exterior (or exterior or body) and interior of a vehicle. Accordingly, the 3D reconstruction system (1) can perform a scan of the exterior of the vehicle, a scan of the interior of the vehicle, and a scan of vehicle parts, and generate 3D depth information about the vehicle based on the results of the scan of the exterior of the vehicle, the scan of the interior of the vehicle, and the scan of vehicle parts.

[0075] The 3D reconstruction system (1) can perform a scan when a vehicle and vehicle parts are located in a preset space (or area). For example, the 3D reconstruction system (1) can perform a scan of a vehicle based on its location while tracking the vehicle's location.

[0076] According to an embodiment, the 3D reconstruction system (1) can perform a scan when a vehicle is positioned adjacent to the outside of an entrance of a work cell (WC), when a vehicle enters and is positioned inside a work cell through the entrance, when a vehicle is positioned in a preset area (e.g., an internal scan work area) within a work cell, or when a vehicle parts receiving container (e.g., a pallet) is positioned in a preset area (or space).

[0077]

[0078] Referring to FIGS. 1 and 2, a 3D reconstruction system (1) according to an embodiment may include a positioning device (100), a scanning device (200), a 3D depth information generation device (300), a controller (400), and a storage unit (500), but the configuration of the 3D reconstruction system (1) is not limited thereto.

[0079] The position confirmation device (100) can confirm the position of the vehicle (V), which is the scan target, and generate information on the position and attitude of the vehicle (V). For example, the position confirmation device (100) can generate information on the position and attitude of the vehicle (V), transmit it to the controller (400), and store it in the storage unit (500).

[0080] A 3D reconstruction system (1) according to an embodiment performs a scan of the interior of a vehicle (V), and in order to provide a space in which the interior of the vehicle (V) can be scanned, the upper part of the vehicle (V) includes an open area.

[0081] The position confirmation device (100) can confirm the position and posture of a vehicle (V) entering a workspace (WC) in real time. For example, the vehicle (V) may enter the workspace (WC) via a conveyor. In some embodiments, the vehicle (V) may enter the workspace (WC) while mounted on a moving body moved by the conveyor.

[0082] According to an embodiment, the position confirmation device (100) may include a tag (110) installed in a vehicle (V) and providing its own position information, and a position information generation unit (120) installed in a work space (WC) and generating information on the position and attitude of the vehicle (V) based on the position information from the tag (110).

[0083] For example, the tag (110) can provide 3D coordinate information (X, Y, Z) as its own location information, and the location information generation unit (120) can be installed in a fixed form within the work space (WC).

[0084] For example, the location confirmation device (100) may include a plurality of tags (110) and a plurality of location information generation units (120), and may be implemented as an ultra-wideband (UWB)-based system, an ultrasound-based system, a laser triangulation-based system, etc.

[0085] According to an embodiment, the information generated by the positioning device (100) can be used to generate an optimal trajectory for scanning and collision avoidance.

[0086] A scanning device (200) can scan a vehicle (V) and vehicle parts and generate (or output) scan results (or scan images). The scan results can be provided to a 3D depth information generating device (300) and stored in a storage unit (500).

[0087] According to an embodiment, the scanning device (200) can scan the exterior (or exterior or body) and interior of the vehicle (V).

[0088] A scanning device (200) according to an embodiment can perform a scan of the exterior of a vehicle, a scan of the interior of a vehicle, and a scan of vehicle parts, and can generate a vehicle exterior scan result, a vehicle interior scan result, and a vehicle parts scan result.

[0089] According to an embodiment, the scanning device (200) may include a first scanning device (210), a second scanning device (220), and a third scanning device (230).

[0090] FIG. 3 is a drawing showing an example in which a scanning device (200) according to an embodiment of the present invention scans the exterior of a vehicle (V) located in a first scan area (SR1).

[0091] As illustrated in FIG. 3, the first scanning device (210) can scan the exterior of a vehicle (V) located within a first scanning area (SR1) set adjacent to the outside of the entrance of the work space (WC).

[0092] For example, the first scanning device (210) may be installed adjacent to the entrance of the work space (WC) so as to face the first scan area (SR1). For example, the first scanning device (210) may be installed outside or inside the entrance of the work space (WC) so as to face the first scan area (SR1). For example, the first scanning device (210) may be fixedly installed at the entrance of the work space (WC) so as to scan the vehicle (V) from a certain direction.

[0093] Here, fixed installation may mean that the first scan area for the first scan device (210) does not change depending on the movement of the vehicle (V) during scanning.

[0094] For example, depending on system reset, size of vehicle (V), etc., the first scan area (SR1) for the first scan device (210) may change before the scanning operation, but once the scanning operation is performed, the first scan area (SR1) may not change until the scanning operation is completed.

[0095] According to an embodiment, the first scan device (210) may include, but is not limited to, a first-first scan device (211) and a first-second scan device (212) installed respectively at the upper two edges of the entrance to the work space (WC).

[0096] For example, the first scanning device (210) may include three or more scanning devices that scan the exterior of the vehicle (V) within the first scan area (SR1) at different locations.

[0097] In this way, by scanning the exterior of the vehicle (V) from different directions using a plurality of first scanning devices (211, 212), a more accurate scan result can be generated than when scanning the exterior of the vehicle (V) using a single scanning device.

[0098] For example, the first-first scan device (211) and the first-second scan device (212) may include a scan sensor (SS1) (SS). For example, the first-first scan device (211) and the first-second scan device (212) may be in the form of a robot arm on which the scan sensor (SS1) is installed.

[0099] According to an embodiment, the first scanning device (210) can perform a scanning operation in response to a trigger signal from the controller (400).

[0100] FIG. 4 is a drawing showing an example in which a scanning device (200) according to an embodiment of the present invention scans the exterior and interior of a vehicle (V) located across first to third scan areas (SR1 to SR3).

[0101] As illustrated in FIG. 4, the second scanning device (220) can scan the exterior of the vehicle (V) located within the second scanning area (SR2) set adjacent to the entrance inside the work space (WC).

[0102] When the vehicle (V) is positioned across the first scan area (SR1) and the second scan area (SR1), both the first and second scan devices (220) can perform a scanning operation.

[0103] For example, the second scanning device (220) may be installed adjacent to the entrance inside the work space (WC) so as to face the second scan area (SR2). Depending on the embodiment, the second scanning device (220) may also be installed outside the entrance of the work space (WC) so as to face the second scan area (SR2). For example, the second scanning device (220) may be fixedly installed so as to scan the vehicle (V) from a certain direction.

[0104] Here, being fixedly installed may mean that the second scan area (SR2) for the second scan device (220) does not change depending on the movement of the vehicle (V) during scanning.

[0105] For example, depending on system reset, size of vehicle (V), etc., the second scan area (SR2) for the second scan device (220) may change before the scanning operation, but once the scanning operation is performed, the second scan area (SR2) may not change until the scanning operation is completed.

[0106] According to an embodiment, the second scan device (220) may include, but is not limited to, a second-first scan device (221) and a second-second scan device (222) installed respectively at the upper two edges of the entrance to the work space (WC).

[0107] For example, the second scanning device (220) may include three or more scanning devices that scan the exterior of the vehicle (V) within the second scan area (SR2) at different locations.

[0108] In this way, by scanning the exterior of the vehicle (V) from different directions using a plurality of second scanning devices (221, 222), a more accurate scan result can be generated than when scanning the exterior of the vehicle (V) using a single scanning device.

[0109] For example, the 2-1 scan device (221) and the 2-2 scan device (222) may include a scan sensor (SS2) (SS). As an example, the 2-1 scan device (221) and the 2-2 scan device (222) may include a robot arm on which the scan sensor (SS2) is installed.

[0110] In this way, the scanning device (200) includes a first scanning device (210) and a second scanning device (220) that scan the exterior of the vehicle (V) in different directions for different areas (SR1, SR2), thereby enabling scanning the exterior of the vehicle (V) from various angles. Accordingly, it is possible to generate 3D depth information for the vehicle (V) based on images scanned from various angles, thereby increasing the completeness of the 3D depth information.

[0111] According to an embodiment, the second scanning device (220) can perform a scanning operation in response to a trigger signal from the controller (400).

[0112] As illustrated in FIG. 4, the third scan device (230) can scan the interior of a vehicle (V) located within a preset third scan area (SR3) within a work space (WC). Here, the third scan area (SR3) can overlap with the second scan area (SR2), and when the vehicle (V) is located across the second and third scan areas (SR2, SR3), the second scan device (220) and the third scan device (230) can perform a scanning operation together for a predetermined period of time.

[0113] According to an embodiment, the third scanning device (230) can be implemented in a movable form and can be rotated in various directions so as to scan the interior of the vehicle (V) in detail.

[0114] For example, the third scanning device (230) can be installed on a conveyor (C) provided at the top within the work space (WC), and can scan the interior of the vehicle (V) while moving in accordance with the movement of the conveyor (C).

[0115] For example, the third scanning device (230) and the conveyor (C) can operate under the control of the controller (400).

[0116] One or more third scanning devices (230) may be installed within the work space (WC), and in this embodiment, two third scanning devices (231, 232) are installed within the work space (WC).

[0117] According to an embodiment, conveyors (C1, C2) are installed at the upper two edges of the work space (WC) in parallel with the vehicle entry direction, and a third scanning device (231, 232) can be installed on each of the two conveyors (C1, C2).

[0118] Of course, the installation structure of the conveyor (C) is not limited to this, and for example, the conveyor (C) may be installed in a direction crossing the vehicle suppression direction, or may be installed in an X shape.

[0119] According to this arrangement, the third scanning device (230) can move in accordance with the movement of the conveyor (C) and scan the interior of the vehicle (V) in more detail.

[0120] According to an embodiment, the third scanning device (230) may include a multi-joint robot arm equipped with a scan sensor (SS3) (SS).

[0121] In this way, as the third scanning device (230) including the multi-joint robot arm moves on the conveyor (C), the scan sensor (SS3) of the third scanning device (230) can be moved to a position where high-resolution scanning is possible.

[0122] According to an embodiment, the third scanning device (230) can scan the interior of the vehicle (V) while moving along an optimal trajectory generated for scanning and collision avoidance based on information about the position and attitude of the vehicle (V).

[0123] For example, the third scanning device (230) can generate a scan trajectory based on information about the position and attitude of the vehicle (V) and scan the interior of the vehicle (V) based on the generated scan trajectory. For example, the third scanning device (230) can scan the interior of the vehicle (V) based on scan trajectory information transmitted from the controller (400).

[0124] According to an embodiment, the third scanning device (230) may generate information related to the movement of the robot arm and the end-effector while scanning the interior of the vehicle. The movement information regarding the robot arm and the end-effector may be used to estimate the posture of the scan sensor (SS3) of the third scanning device (230).

[0125] The third scanning device (230) can generate (or output) vehicle interior scan information including a vehicle interior scan image and movement information, and the vehicle interior scan information can be provided to the 3D depth information generating device (300) and stored in the storage unit (500).

[0126] According to an embodiment, the third scan device (230) can scan a vehicle part located in a fourth scan area (SR4) preset within the work space (WC).

[0127] FIG. 5 is a drawing showing an example of a scanning device (200) according to an embodiment of the present invention scanning a vehicle component (VP) located in a fourth scan area (SR4).

[0128] Referring to FIG. 5, the fourth scan area (SR4) may exist in multiple locations within the workspace (WC) and may be located at an edge within the workspace (WC). For example, the fourth scan area (SR4) may be configured so as not to interfere with the movement of the third scan device (230).

[0129] For example, the fourth scan area (SR4) may be formed at a predetermined distance from the top of the conveyor (C). For example, the fourth scan area (SR4) may also be formed on the floor within the work space (WC).

[0130] According to an embodiment, a vehicle parts container (e.g., pallet) (P) containing vehicle parts (VP) may be placed in the fourth scan area (SR4). For example, the vehicle parts container (P) may be placed in the fourth scan area (SR4) by a separate transport system.

[0131] For example, the transport system may provide count information to the controller (400) when placing a vehicle parts container (P) in the fourth scan area (SR4) or retrieving a vehicle parts container (P) located in the fourth scan area (SR4).

[0132] For example, when the transport system places a vehicle parts container (P) in the fourth scan area (SR4), it may count up (+1) from the current count information, and when the transport system retrieves the vehicle parts container (P) in the fourth scan area (SR4), it may count down (-1) from the current count information, and then provide the updated count information to the controller (400).

[0133] The third scanning device (230) can generate information related to the movement of the robot arm and end-effector while scanning the vehicle component (VP). The movement information regarding the robot arm and end-effector can be used to estimate the posture of the scan sensor (SS3) of the third scanning device (230).

[0134] The third scanning device (230) can generate (or output) vehicle part scan information including a vehicle part scan image and movement information, and the vehicle part scan information can be provided to the 3D depth information generating device (300) and stored in the storage unit (500).

[0135] The 3D depth information generation device (300) can generate 3D depth information based on the scan result (or scan image) generated by the scanning device (200). The 3D depth information generated by the 3D depth information generation device (300) can be provided to the controller (400) and stored in the storage unit (500).

[0136] According to an embodiment, a 3D depth information generating device (300) can generate 3D depth information for a vehicle (V) based on a vehicle exterior scan image and a vehicle interior scan image, and can generate 3D depth information for a vehicle part (VP) based on a vehicle part scan image.

[0137] According to an embodiment, the 3D depth information generating device (300) can generate 3D depth information for the vehicle (V) by considering movement information for the robot arm and the end-effector related to the vehicle interior scan image.

[0138] According to an embodiment, a 3D depth information generating device (300) can generate 3D depth information for a vehicle component (VP) by considering movement information for a robot arm and an end-effector related to a vehicle component scan image.

[0139] The controller (400) can receive various information generated (or output) from components within the 3D reconstruction system (1) and perform control over the components within the 3D reconstruction system (1).

[0140] According to an embodiment, the controller (400) may output a trigger signal for controlling the scanning device (200) based on vehicle (V)-related position information. For example, the controller (400) may output a trigger signal for controlling the first and second scanning devices (210, 220) based on vehicle (V)-related position information.

[0141] If the controller (400) determines that the vehicle (V) is located in the first scan area based on the vehicle (V)-related location information, it can output a trigger signal to the first scan device (210).

[0142] And, if the controller (400) determines that the vehicle (V) is located in the second scan area based on the vehicle (V)-related location information, it can output a trigger signal to the second scan device (220).

[0143] According to an embodiment, the controller (400) can control the scanning device (200) based on the vehicle (V) related position and attitude information generated from the position confirmation device (100).

[0144] The controller (400) can generate a scan trajectory for controlling the scanning device (200) based on position and attitude information related to the vehicle (V).

[0145] For example, the controller (400) can generate a scan trajectory for controlling the third scanning device (230) based on position and attitude information related to the vehicle (V).

[0146] Additionally, the controller (400) can control the conveyor (C) while simultaneously controlling the third scan device (230) based on the scan trajectory.

[0147] That is, the controller (400) can control the third scan device (230) and the conveyor (C) to enable the scan sensor (SS) of the third scan device (230) to scan the interior of the vehicle (V) in detail.

[0148] According to an embodiment, the controller (400) may control the third scan device (230) to perform a scan on a vehicle component (VP) when a preset condition is satisfied, such as when the controller (400) determines that the robot arm of the third scan device (230) is in an idle state based on movement information of the robot arm / end-effector, when the controller determines that the vehicle (V) has left the scan area based on position information related to the vehicle, when the controller determines that a new vehicle component receiving container (P) is located in the fourth scan area (SR4) based on count information from the transport system, or when there is a request from a higher system.

[0149] According to an embodiment, the controller (400) can compare 3D depth information with an original CAD model and calculate a change in the position of the mounting points of the vehicle (V) and vehicle components (VP).

[0150] For example, the controller (400) can calculate a position change for a mounting point of a vehicle (V) / vehicle component (VP) using a subtraction method. For example, the controller (400) can calculate a position change for a mounting point of a vehicle (V) / vehicle component (VP) based on six DOF (Degrees of Freedom) transformation coordinates and rotations (Roll, Pitch, Yaw).

[0151] Information on the change in position of a mounting point related to a vehicle (V) / vehicle component (VP) generated by the controller (400) can be provided to a higher system and stored in a storage unit (500).

[0152] Information on position changes in mounting points of vehicles / vehicle components can be used as manufacturing statistics and can be useful for automated robot teaching.

[0153] The controller (400) can control a conveyor that moves the vehicle (V) into the work space (WC) or moves the vehicle (V) out of the work space (WC).

[0154]

[0155] FIG. 6a is a front view of a scan sensor (SS) included in a scan device (200) according to an embodiment of the present invention, FIG. 6b is a schematic view of the scan sensor (SS) as seen from direction A of FIG. 6a, and FIG. 6c is a schematic view of the scan sensor (SS) as seen from direction B of FIG. 6a.

[0156] Referring to FIGS. 6A to 6C, a scan sensor (SS) according to an embodiment of the present invention can irradiate light and scan an object based on the reflected light that is reflected back by the irradiated light from the object.

[0157] According to an embodiment, the scan sensor (SS) may be configured to include a light source (1100), a camera (1200), a first polarizer (1300), a second polarizer (1400), and a distance sensor (1500), but the configuration of the scan sensor (SS) is not limited thereto.

[0158] For example, a light source (1100), a camera (1200), a first polarizer (1300), a second polarizer (1400), and a distance sensor (1500) can be placed in the main body (1600).

[0159] The light source (1100) is a configuration that irradiates light, and may be, for example, an LED (light emitting diode) that irradiates infrared or visible light, but is not limited thereto.

[0160] A plurality of light sources (1100) can be arranged in the main body (1600), and when a plurality of light sources (1100) are arranged, the plurality of light sources (1100) can be arranged in a manner of being aligned in one direction (X).

[0161] The camera (1200) can be placed adjacent to the side of the light source (1100) and can obtain an image of the object by receiving light (reflected light) that is reflected back from the object by the light irradiated from the light source (1100).

[0162] The camera (1200) forms a pair with a light source (1100) and can be placed adjacent to the pair of light sources (1100).

[0163] A plurality of cameras (1200) can be arranged on the main body (1600), and when a plurality of cameras (1200) are arranged, the plurality of cameras (1200) can be arranged in a manner of being aligned in one direction (X).

[0164] A pair of light sources (1100) and cameras (1200) can be arranged in alignment in one direction (X) and the other direction (Y) perpendicular thereto.

[0165] The first polarizer (1300) can be placed at a predetermined distance in front of the light source (1100) and can prevent glare caused by light irradiated from the light source (1100).

[0166] When a plurality of light sources (1100) are arranged in the main body (1600), a plurality of first polarizers (1300) are also arranged accordingly, and the plurality of first polarizers (1300) can be arranged in a manner of being aligned in one direction (X).

[0167] The second polarizer (1400) can be placed at a predetermined distance in front of the camera (1200) and can filter natural light reflected from an object and entering the camera (1200).

[0168] When a plurality of cameras (1200) are arranged in the main body (1600), a plurality of second polarizers (1400) are also arranged accordingly, and the plurality of second polarizers (1400) can be arranged in a manner of being aligned in one direction (X).

[0169] The distance sensor (1500) can measure the distance to an object, and can measure the distance using a TOF (Time of Flight) method that can measure the distance even for objects at a long distance, but is not limited thereto.

[0170] For example, the distance sensor (1500) can be arranged aligned with the camera (1200) in the other direction (Y).

[0171]

[0172] FIG. 7 is a drawing for explaining a 3D reconstruction method according to an embodiment of the present invention.

[0173] The step-by-step operations illustrated in FIG. 7 can be performed by the 3D reconstruction system described with reference to FIGS. 1 to 6c.

[0174] Referring to FIGS. 1 to 7, a 3D reconstruction method according to an embodiment of the present invention is described, assuming that a vehicle (V) being a work target is moving to a work space (WC).

[0175] First, as the vehicle (V) moves to the work space (WC), the positioning device (100) can check the position of the vehicle (V) in real time and generate position and attitude information related to the vehicle (V) in real time (S700).

[0176] Generating vehicle (V)-related position and attitude information may be continuously performed until the scan of the vehicle (V) is completed. For example, vehicle (V)-related position and attitude information may be provided to the controller (400) and stored in the storage unit (500).

[0177] When the vehicle (V) is positioned in the first scan area (SR1), the first scan device (210) can scan the exterior of the vehicle (V) (S710).

[0178] In step S710, the first scanning device (210) can perform a scan on the vehicle (V) in response to a trigger signal from the controller (400).

[0179] A scan image (first vehicle exterior scan image) generated by scanning of the first scan device (210) can be provided to a 3D depth information generation device (300) and stored in a storage unit (500).

[0180] When the vehicle (V) is positioned in the second scan area (SR2), the second scan device (220) can scan the exterior of the vehicle (V) (S720).

[0181] In step S720, the second scanning device (220) can perform a scan on the vehicle (V) in response to a trigger signal from the controller (400).

[0182] A scan image (second vehicle exterior scan image) generated by scanning of the second scan device (220) can be provided to the 3D depth information generation device (300) and stored in the storage unit (500).

[0183] When the vehicle (V) is located in the third scan area (SR3), the third scan device (230) can scan the interior of the vehicle (V) (S730).

[0184] In step S730, the third scanning device (230) can generate a scan trajectory based on the vehicle (V) related position and attitude information generated by the position confirmation device (100), and scan the interior of the vehicle (V) along the generated scan trajectory.

[0185] In step S730, the third scanning device (230) can scan the interior of the vehicle (V) according to scan trajectory information generated by the controller (400) based on the vehicle (V) related position and attitude information generated by the position confirmation device (100).

[0186] In step S730, the third scanning device (230) can scan the interior of the vehicle (V) while moving in accordance with the operation of the conveyor (C) on which it is installed.

[0187] A scan image (vehicle interior scan image) generated by scanning of the third scan device (230) can be provided to the 3D depth information generation device (300) and stored in the storage unit (500).

[0188] In step S730, the third scanning device (230) can generate information related to the movement of the robot arm and the end-effector while scanning the interior of the vehicle (V).

[0189] The movement information of the robot arm / end-effector generated by the third scanning device (230) can be provided to the 3D depth information generating device (300) and stored in the storage unit (500).

[0190] Thereafter, the third scan device (230) can perform a scan on a vehicle component (VP) located in the fourth scan area (SR4) (S740).

[0191] In step S740, the third scanning device (230) can perform a scan on a vehicle component (VP) in response to an instruction from the controller (400).

[0192] According to an embodiment, the controller (400) may output a vehicle parts scan instruction signal to the third scan device (230) when it determines that the robot arm of the third scan device (230) is in an idle state based on movement information of the robot arm / end-effector.

[0193] According to an embodiment, the controller (400) may output a vehicle component scan instruction signal to the third scan device (230) when it determines that the vehicle has left the scan area (first to third scan areas) based on vehicle (V)-related location information.

[0194] According to an embodiment, the controller (400) may output a vehicle part scan instruction signal to the third scan device (230) when it determines that a new vehicle part (VP) is located in the fourth scan area (SR4) based on count information from a transport system that transports the vehicle part (VP) into the work space (WC).

[0195] According to an embodiment, the controller (400) can output a vehicle parts scan instruction signal to the third scan device (230) when requested by the upper system.

[0196] A scan image (vehicle parts scan image) generated by scanning of the third scan device (230) can be provided to the 3D depth information generation device (300) and stored in the storage unit (500).

[0197] In step S740, the third scanning device (230) can generate information related to the movement of the robot arm and the end-effector while scanning the vehicle component (VP).

[0198] The movement information of the robot arm / end-effector generated by the third scanning device (230) can be provided to the 3D depth information generating device (300) and stored in the storage unit (500).

[0199] When scanning of a vehicle (V) and a vehicle part (VP) is completed according to steps S710 to S740, the 3D depth information generation device (300) can generate 3D depth information based on the scanned image (S750).

[0200] In step S750, the 3D depth information generating device (300) can generate 3D depth information for a vehicle (V) based on a vehicle exterior scan image and a vehicle interior scan image, and can generate 3D depth information for a vehicle part (VP) based on a vehicle part scan image.

[0201] In step S750, the 3D depth information generation device (300) can generate 3D depth information for the vehicle (V) by considering movement information related to the robot arm / end-effector related to the vehicle interior scan image.

[0202] In step S750, the 3D depth information generation device (300) can generate 3D depth information for a vehicle component (VP) by considering movement information related to a robot arm / end-effector associated with a vehicle component scan image.

[0203] 3D depth information generated by the 3D depth information generation device (300) can be provided to the controller (400) and stored in the storage unit (500).

[0204] Thereafter, the controller (400) can compare the 3D depth information with the original CAD model and calculate the change in position of the mounting points of the vehicle (V) and vehicle parts (VP) (S760).

[0205] In step S760, the controller (400) may calculate a position change for the mounting point of the vehicle (V) / vehicle component (VP) using a subtraction method. For example, the controller (400) may calculate a position change for the mounting point of the vehicle (V) / vehicle component (VP) based on six DOF (Degrees of Freedom) transformation coordinates and rotations (Roll, Pitch, Yaw).

[0206] Information on the change in position of a mounting point related to a vehicle (V) / vehicle component (VP) generated by the controller (400) can be provided to a higher system and stored in a storage unit (500).

[0207] According to an embodiment of the present invention, since both scanning of a vehicle (V) and scanning of a vehicle part (VP) can be performed in one work space (WC), the time required to scan a vehicle (V) / vehicle part (VP) can be shortened.

[0208] In addition, since the movement information related to the robot arm / end-effector used to estimate the posture of the scan sensor (SS) of the third scan device (230) is acquired in real time while scanning the interior of the vehicle (V) and the vehicle parts (VP), the time required to estimate the posture of the scan sensor (SS) of the third scan device (230) can be shortened.

[0209] In addition, since it is possible to obtain position change information for mounting points of vehicles (V) / vehicle parts (VP) by comparing 3D depth information generated based on scanned images with the original CAD model, it is possible to build a tolerance model for manufacturing errors in parts and processes.

[0210]

[0211] Although the embodiments of the present invention have been described in more detail with reference to the attached drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be implemented without departing from the technical spirit of the present invention. Therefore, the embodiments disclosed in this specification are not intended to limit the technical spirit of the present invention, but to explain it, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention should be interpreted by the claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present invention.

[0212] [Explanation of symbols]

[0213] 1: 3D reconstruction system 100: Positioning device

[0214] 110: Tag 120: Location information generation unit

[0215] 200: Scanning device 210: First scanning device

[0216] 220: Second scanning device 230: Third scanning device

[0217] 300: 3D depth information generation device 400: Controller

[0218] 500: Storage

Claims

1. A positioning device that identifies the location of a vehicle entering the work space; A scanning device for scanning the interior and exterior of a vehicle entering the work space, while scanning vehicle parts located within the work space; A 3D depth information generating device that generates 3D depth information for the vehicle and the vehicle parts based on the scan image generated by the scanning device; and A 3D reconstruction system, comprising a controller that controls the scanning device based on information generated by the position confirmation device and information generated by the scanning device.

2. In paragraph 1, The above location confirmation device, A tag installed in the above vehicle and providing its own location information; and A 3D reconstruction system, comprising a location information generation unit that generates location and attitude information for the vehicle based on the location information provided from the tag.

3. In paragraph 1, A 3D reconstruction system, wherein the controller compares the 3D depth information generated by the 3D depth information generating device with an original CAD model and calculates a change in the position of mounting points of the vehicle and the vehicle parts.

4. In paragraph 1, The above scanning device, A first scanning device for scanning the exterior of the vehicle located in a first scanning area set outside the entrance of the work space; A second scanning device for scanning the exterior of the vehicle, positioned in a second scanning area adjacent to the entrance of the working space within the working space; and A 3D reconstruction system comprising a third scan device for scanning the interior of the vehicle located in a third scan area set within the work space, while scanning the vehicle component located in a fourth scan area within the work space.

5. In paragraph 4, A 3D reconstruction system, wherein the first and second scan devices include robot arms equipped with scan sensors, and the third scan device includes a multi-joint robot arm including a scan sensor.

6. In paragraph 4, The above scanning device comprises a plurality of first scanning devices, A 3D reconstruction system, wherein said plurality of first scan devices scan the exterior of said vehicle within said first scan area at different locations.

7. In paragraph 4, The above scanning device comprises a plurality of second scanning devices, A 3D reconstruction system, wherein said plurality of second scan devices scan the exterior of said vehicle within said second scan area at different locations.

8. In paragraph 4, A 3D reconstruction system in which the first to third scanning devices perform a scanning operation alone or together, depending on the position of the vehicle.

9. In paragraph 4, A 3D reconstruction system, wherein a conveyor is installed within the work space, and the third scanning device is installed on the conveyor.

10. In paragraph 4, A 3D reconstruction system in which the third scanning device generates a scan trajectory based on the vehicle-related position and attitude information generated by the position confirmation device, and scans the interior of the vehicle while moving along the scan trajectory.

11. In paragraph 4, A 3D reconstruction system in which the third scan device scans the interior of the vehicle while moving according to scan trajectory information provided from the controller.

12. In paragraph 4, A 3D reconstruction system, wherein the controller controls the third scanning device to scan the vehicle component when it determines that the vehicle has left the preset scan area based on the location information about the vehicle generated by the position confirmation device.

13. In paragraph 5, A 3D reconstruction system in which the third scanning device scans the interior of the vehicle while generating movement information for the robot arm and end-effector.

14. In paragraph 5, A 3D reconstruction system in which the third scanning device generates movement information for the robot arm and end-effector while scanning the vehicle component.

15. In paragraph 12, A 3D reconstruction system, wherein the controller controls the third scan device to scan the vehicle component when it determines that the robot arm is in an idling state based on movement information generated by the third scan device while scanning the interior of the vehicle.

16. A step of confirming the location of a vehicle entering the work space using a position confirmation device; A step of scanning the interior and exterior of the vehicle entering the work space using a scanning device; A step of scanning a vehicle part located within the above work space using the above scanning device; and A 3D reconstruction method, comprising the step of generating 3D depth information for the vehicle and the vehicle parts using a 3D depth information generating device based on a scan image generated by the scanning device.

17. In paragraph 16, The step of scanning the interior and exterior of the above vehicle using a scanning device is: A step of scanning the exterior of the vehicle located in a first scan area set outside the entrance of the work space using a first scan device; A step of scanning the exterior of the vehicle located in a second scan area set adjacent to the entrance of the work space within the work space using a second scan device; and A 3D reconstruction method, comprising the step of scanning the interior of the vehicle located in a third scan area set within the above working space using a third scan device.

18. In paragraph 17, A 3D reconstruction method, wherein the step of scanning using the third scanning device includes generating movement information for a multi-joint robot arm and an end-effector included in the third scanning device while scanning the interior of the vehicle.

19. In paragraph 16, The step of scanning the above vehicle parts using the above scanning device is: A 3D reconstruction method comprising scanning the vehicle component located in a fourth scan area set within the above working space using a third scan device.

20. In paragraph 19, The step of scanning the above vehicle parts using the above scanning device is: A 3D reconstruction method comprising generating movement information for a multi-joint robot arm and an end-effector included in the third scanning device while scanning the vehicle component.

21. In paragraph 16, A 3D reconstruction method, comprising the steps of comparing the 3D depth information with an original CAD model and calculating a change in position of mounting points of the vehicle and the vehicle components.

Images