Augmented reality system of stamping die and control method thereof

Abstract

An augmented reality (AR) system includes a data generation section configured to generate three-dimensional (3D) data corresponding to a stamping die product, and an AR terminal configured to convert the 3D data into AR data. The AR terminal includes a data input section configured to receive 3D data and display information corresponding to position information of each node of the 3D data. The AR terminal further includes a data processing section configured to convert the 3D data into the form of a polyhedral mesh according to the form information type of the 3D data, and generate AR data in the form of a 3D virtual model by matching with the display information. The AR terminal also includes a data enhancement section configured to enhance and match AR data on a physical product displayed on a camera image, and enhance and display display information of each node through a user interface (UI).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to Korean Patent Application No. 10-2023-0117000, filed on September 4, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to an augmented reality (AR) system for a press mold and a control method thereof. Background Art

[0004] Generally speaking, augmented reality (AR) is a computer graphics technology used to synthesize virtual information in an actual existing engineering environment to make virtual objects look like real objects.

[0005] In the existing manufacturing sector, most AR technologies are related to work instructions using two-dimensional (2D) data.

[0006] For the marker-based scan data enhancement technology, the position and posture of the stamping die product can be identified using a marker-based coordinate system recognized in the camera image, and the three-dimensional (3D) data can be enhanced.

[0007] However, when there are no markers on the camera screen or there are recognition errors (e.g., wrong shooting distance, wrong focus, etc.), it will cause data augmentation to fail, resulting in a decrease in stability.

[0008] Although the markerless enhancement technology can solve the shortcomings brought by the above-mentioned markers, the position and posture recognition accuracy of the physical product is poor, and it cannot meet the consistency between the test samples required by the mold department and the data enhancement required by the mold department.

[0009] The description of the background art is intended to increase the understanding of the background of the present invention and may include non-prior arts already known to those of ordinary skill in the art to which the present invention relates. Summary of the invention

[0010] As mentioned above, in the existing manufacturing sector, most augmented reality (AR) technologies are related to work instructions using two-dimensional (2D) data. In the prior art, enhanced three-dimensional (3D) engineering virtual data (e.g., obtained through forming analysis, scanning, etc.) for assisting stamping die related work has not yet been introduced.

[0011] The present invention provides an AR system for stamping dies and a control method thereof, wherein 3D data of products used by the stamping die department are processed into a lightweight optimized state and enhanced to obtain AR data, and the 3D coordinate system is corrected by marker tracking to improve consistency with the physical product.

[0012] According to one aspect of the present invention, an augmented reality (AR) system for a stamping die is provided. The AR system includes a data generation unit configured to generate three-dimensional (3D) data corresponding to a stamping die product. The AR system also includes an AR terminal configured to convert the 3D data into AR data to be in the real world, and match the AR data with a physical product. The AR terminal includes a data input unit configured to receive the 3D data generated by the data generation unit and display information corresponding to the position information of each node of the 3D data. The AR terminal also includes a data processing unit configured to convert the 3D data into a polyhedral mesh form according to the type of form information of the 3D data, and generate AR data in the form of a 3D virtual model by matching with the display information. The AR terminal also includes a data enhancement unit configured to enhance and match the AR data on the physical product displayed on the camera image, and enhance and display the display information of each node of the 3D data through a user interface (UI). The display information of each node of the 3D data may be information collected by molding analysis of the physical product. The AR terminal also includes a control unit configured to control the overall operation of the components applied to the AR terminal.

[0013] The data generation unit may include a data generation module configured to generate 3D data, including at least one of 3D engineering data generated by a user modeling a stamping die product, coordinate measuring machine (CMM) data generated by measuring the state of a physical product, or 3D scanning data generated by scanning a physical product. The data generation unit may also include a display information generation module configured to generate display information by collecting at least one property information of the position of each node, hole processing state, curved surface, step, size, thickness and molding analysis information according to the shape of the physical product.

[0014] The data processing unit may include a data conversion module configured to identify the type of form information of the 3D data and convert the 3D data into a graphic mesh form when the 3D data is not in a graphic mesh form. The data processing unit may also include a data lightweight module (data reduction module) configured to achieve lightweight by reducing the number of grids input to the graphic mesh. The data processing unit may also include a display information matching module configured to match the display information of each node according to the position information of the shape of the lightweight graphic mesh, thereby generating AR data to be enhanced in the form of a 3D virtual model in the real world.

[0015] The data conversion module can be configured to identify the type of form information of the 3D data, and when the 3D data is in 3D computer-aided design (CAD) form, convert the 3D data into a graphic mesh form of a polyhedral structure, and when the 3D data is in a graphic mesh form, skip a separate data conversion process.

[0016] The data enhancement unit may include a space recognition module configured to convert a real-world workspace input through a camera image into a virtual 3D space. The data enhancement unit may also include a workspace generation module configured to generate a 3D matching coordinate system based on at least one marker identified by the camera image. The data enhancement unit may also include an AR data enhancement module configured to load AR data into the real world and enhance and match the AR data with the position and posture of the product based on the 3D matching coordinate system. The data enhancement unit may also include a display information enhancement module configured to enhance the display information corresponding to the position information of each node of the product through the UI when the AR data is enhanced on the physical product.

[0017] The AR data enhancement module can be configured to generate a matching coordinate system based on the added markers, and use the matching coordinate system to correct the position and posture of the product by correcting the 3D matching coordinate system based on spatial recognition.

[0018] The display information enhancement module may be configured to enhance the display information based on the camera image by using a UI for recognizing a user's hand. The UI may include a user input menu for displaying at least one of an experiment, analysis, inspection, and evaluation of product formability.

[0019] The display information enhancement module can be configured to select a user input menu according to information input through the UI, and display at least one product trend of thickness reduction rate, panel sag rate, and plastic strain rate for each node corresponding to AR data matching the product through a color graph.

[0020] The UI may include a first UI configured to recognize a left hand of a user to check a data list to be enhanced, and a second UI configured to recognize a right hand of the user to select a desired data list and click a numerical value to be viewed.

[0021] According to another aspect of the present invention, a control method for an AR terminal applied to a stamping die process is provided. The control method includes receiving 3D data corresponding to the stamping die product input input from a data generation unit, and display information generated for each node of the product by analyzing the 3D data. The control method also includes converting the received 3D data into the form of a graphic grid according to the type of form information of the 3D data. The control method also includes generating AR data in the form of a 3D virtual model by matching with the display information, and matching the AR data with the physical product displayed by the camera image. The control method also includes enhancing and displaying the display information corresponding to the position information of each node of the product through the UI when the AR data is matched.

[0022] Generating the AR data may include identifying a type of form information of the 3D data to determine whether the 3D data is in a graphic mesh form, and converting the 3D data into a graphic mesh form of a polyhedron shape when the type of form information of the 3D data is not in a graphic mesh form.

[0023] Generating the AR data may further include lightweighting the graphic mesh data obtained by converting the 3D data by reducing the number of meshes of the graphic mesh data for graphic processing, and generating the 3D virtual model by matching the lightweight graphic mesh data with the display information.

[0024] Matching the AR data may include converting a real-world workspace input through the camera image into a virtual 3D space and generating a 3D matching coordinate system based on pre-formed markers in the workspace. Matching the AR data may also include matching the AR data with the position and posture of the product based on the 3D matching coordinate system by tracking the position of at least one marker on the camera image and loading the AR data into the real world.

[0025] Matching the AR data with the position and posture of the product may include: whenever a marker is added to an image of a camera that moves with the user's line of sight, generating a matching coordinate system based on the added marker, and using the matching coordinate system to correct the position and posture of the product by correcting a 3D matching coordinate system based on spatial recognition.

[0026] Matching the AR data to the position and pose of the product may include enhancing and matching the AR data based on pre-designated coordinates of at least one detection port of the physical product.

[0027] Enhancing and displaying the display information may include selecting a display menu according to information input through the UI based on recognition of a user's hand. Enhancing and displaying the display information may also include displaying a product trend through a color graph, the product trend including at least one of a thickness reduction rate, a panel sag rate, and a plastic strain rate corresponding to position information of each node of the product.

[0028] Enhancing and displaying the display information may include checking a data list to be enhanced through a first UI for identifying a left hand of the user, and selecting a desired data list through a second UI for identifying a right hand of the user and clicking a value to be viewed.

[0029] As described above, according to an embodiment of the present invention, the 3D data used by the stamping die department can be converted into graphic mesh data, and the number of meshes can be reduced to achieve lightweight, thereby reducing the data processing burden and ensuring the responsiveness of AR data.

[0030] Unlike the existing technology, even if all markers are not in the image, the virtual AR data can be matched with the accurate position of the real-world product by identifying the added markers, and the position and posture of the product can be corrected using a matching coordinate system based on spatial recognition.

[0031] In addition, through the UI based on user hand recognition, the analysis information corresponding to the position information of each node of the selected product is enhanced and displayed, allowing users to intuitively check the analysis information of the required parts of the product at the work site. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic block diagram showing a configuration of an augmented reality (AR) system applied to a stamping die process according to an embodiment of the present invention.

[0033] Figure 2 The process of identifying the position and posture of a test sample using a marker-based coordinate system in the prior art is shown.

[0034] Figure 3 is a conceptual diagram of a method for correcting a position and a posture of a product based on space recognition according to an embodiment of the present invention.

[0035] Figure 4 is a schematic flow chart of a control method of an AR system applied to a stamping die process according to an embodiment of the present invention.

[0036] Figure 5 is a view showing an operation state of a user interface (UI) when AR data based on a coordinate measuring machine (CMM) is enhanced on a physical panel according to an embodiment of the present invention.

[0037] Figure 6 It shows a state where actual panel forming analysis results are enhanced in the form of a color map according to an embodiment of the present invention.

[0038] Figure 7 It shows the UI state enhanced on the physical panel based on the panel scanning analysis result according to the embodiment of the present invention.

[0039] Figure 8 FIG. 1 shows a state where panel scan analysis results are enhanced in parallel on a check port according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] Hereinafter, embodiments of the present invention are described in detail with reference to the accompanying drawings so that the embodiments can be implemented by those having ordinary skill in the art.

[0041] The terms used herein are for the purpose of describing the embodiments only and are not intended to limit the present invention.

[0042] As used herein, the singular forms are intended to include the plural forms as well unless the context clearly indicates otherwise.

[0043] It should be understood that when the terms "include" and / or "comprising" are used herein, the presence of the features, integers, steps, operations, constituent elements and / or components is specified, but the presence or addition of one or more other features, integers, steps, operations, constituent elements, components and / or groups thereof is not excluded. As used herein, the term "and / or" includes any one or all combinations of the relevant listed items.

[0044] Throughout the specification, terms such as first, second, A, B, (a) and (b) may be used to describe various components, but the components are not limited by these terms. Such terms are only used to distinguish one component from another, and the nature, order or sequence of the components are not limited by these terms.

[0045] When a component is referred to as being “coupled to” or “connected to” another component, it should be understood that the component may be directly coupled to the other component or may be connected to the other component with another component interposed therebetween.

[0046] In contrast, when a component is referred to as being "directly coupled" or "directly connected" to another component, it should be understood that there are no other components interposed therebetween.

[0047] When a component, device, element or the like of the present invention is described as having a purpose or performing an operation, function, etc., the component, device or element should be considered to be "configured to" satisfy the purpose or perform the operation or function.

[0048] The terms used herein are for describing certain embodiments only and are not intended to limit the present invention.

[0049] As used herein, singular expressions are intended to include plural forms as well unless the context clearly indicates otherwise.

[0050] It should be further understood that one or more of the methods described below or aspects thereof may be performed by at least one control unit.

[0051] The term "control unit" may be understood to include hardware devices such as memory and processors. The memory is configured to store program instructions, and the processor is specifically programmed to execute the program instructions to perform one or more processes described in more detail below. As described herein, the control unit may control the operation of a unit, module, component, device, etc. It should be further understood by a person of ordinary skill in the art that the methods described below may be performed by a device including a control unit and one or more other components.

[0052] Hereinafter, an augmented reality system for a stamping die and a control method thereof according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

[0053] Figure 1 is a schematic block diagram showing a configuration of an augmented reality system applied to a stamping die process according to an embodiment of the present invention.

[0054] See also Figure 1 According to an embodiment of the present invention, an augmented reality (AR) system for a stamping die includes a data generating unit 100 for generating three-dimensional (3D) data corresponding to a stamping die product (including a test sample). The AR system also includes an AR terminal 200 for converting the 3D data into AR data (3D image) for augmenting the 3D data of the real world, matching the AR data with the position of the physical product, and enhancing the display information according to the product analysis.

[0055] The data generation unit 100 generates 3D data according to at least one of the engineer's stamping die product design, 3D dimension measurement, and scanning, and displays the information of each node according to the main shape information of the 3D data. A node can be a point on an element, and an element is given the required degree of freedom to express the behavior of the object to be measured in finite element analysis. Therefore, a node can be understood as a point representing the position information of an element / mesh constituting 3D data and display information about the molding state of the product.

[0056] Hereinafter, the "product" described in the embodiment of the present invention may be a "panel" put into a stamping die in a body shop and formed. However, the embodiment of the present invention is not limited to the panel, but is applicable to molded products in various industrial fields.

[0057] The data generation unit 100 may be implemented by a computing system that can be used by engineers for design, product measurement, product analysis (interpretation), and the like.

[0058] The data generating unit 100 includes a 3D data generating module 110 and a display information generating module 120 .

[0059] The data generation module 110 can generate 3D data, including at least one of 3D engineering data generated by engineers modeling stamping die products, coordinate measuring machine (CMM) data generated by measuring the state (size) of physical products, and 3D scanning data generated by scanning physical products.

[0060] 3D engineering data can be generated in the form of 3D computer-aided design (CAD) by using a design tool (eg, a commercial program). CMM data can be data obtained by measuring the dimensional state of a product by a CMM device with a gauge as a reference.

[0061] 3D scan data can be generated in the form of a graphical mesh based on the physical product.

[0062] The display information generation module 120 generates display information corresponding to the position information of each node of the 3D data generated by the data generation module 110. The display information includes attribute information such as the position of each node, hole processing status, curved surface, step, size (area), thickness, etc. according to the product shape, and molding analysis information of the physical product. The attribute information may include the results of comparing and analyzing the reference attribute information optimized for mass production of the product and the measured attribute information.

[0063] The AR terminal 200 may be an AR-based goggles or glasses-type wearable terminal having at least one RGB depth camera. When the AR terminal 200 is worn on the user's head, it may enhance and match the AR data (i.e., 3D virtual model) corresponding to the position of the real-world physical product displayed on the camera image. In addition, the AR terminal 200 may enhance and display the information of each node collected through the molding analysis of the physical product.

[0064] The display information may include a product inspection menu corresponding to position information of each node during molding analysis or quality inspection of a product produced by a stamping die, and analysis information (interpretation data).

[0065] The display information may be displayed through an enhanced image of each node by linking with a user interface (UI) of the AR terminal 200 .

[0066] The AR terminal 200 includes a data input unit 210 , a data processing unit 220 , a data enhancement unit 230 , and a control unit 240 .

[0067] The data input unit 210 includes a wired / wireless communication device. The data input unit 210 receives the 3D data generated by the data generation unit 100 and the display information corresponding to the position information of each node, and inputs the 3D data and the display information to the data processing unit 220. The input 3D data initially input can be input in the form of 3D CAD or a graphic mesh according to the generation method.

[0068] The data processing unit 220 converts the 3D data input from the data input unit 210 into a polygon mesh format according to the type of shape information, and generates AR data in the form of a 3D virtual model by matching it with the display information.

[0069] The data processing unit 220 may include a data conversion module 221 , a data lightweight module 222 , and a display information matching module 223 .

[0070] The data conversion module 221 identifies whether the type of shape information of the input 3D data is in a graphic mesh form, and if the 3D data is not in a graphic mesh form, the 3D data is converted into a graphic mesh form. For example, when the 3D data is in a 3D CAD form, the data conversion module 221 can convert the 3D CAD into a graphic mesh form of a polyhedral structure.

[0071] However, if the input 3D data is in the form of a graphic mesh, the data conversion module 221 omits a separate data conversion process. In other words, when the data conversion module 221 receives the graphic mesh form as the 3D data, a separate data conversion can be omitted and performed only when the 3D data is not in the form of a graphic mesh.

[0072] The data lightweight module 222 reduces the size of the data (i.e., achieves data lightweight) by reducing the number of grids of the graphic grid input by the data conversion module 221. The graphic grid can be composed of a polyhedral grid structure containing at least one of a triangle and a rectangle. The more grids there are, the heavier the data is, and the more difficult the operation is. Therefore, if there is a delay in the operation processing, it may cause the responsiveness of the AR data to decrease or the image to remain when the user's line of sight moves, causing confusion.

[0073] Therefore, the data lightweight module 222 reduces the size of the number of grids in the graphic grid to optimize the number of grids in the graphic grid, reduce the data processing burden, and ensure enhanced responsiveness.

[0074] The display information matching module 223 generates AR data in the form of a 3D virtual model by matching the display information in the lightweight graphic mesh. The display information matching module 223 can generate AR data to be enhanced in the real world in the form of a 3D virtual model by matching the position information with the display information of each node according to the shape of the lightweight graphic mesh.

[0075] The data enhancement unit 230 enhances and matches the AR data on the physical product image displayed on the RGB depth camera of the AR terminal 200. The data enhancement unit 230 enhances and displays the display information of each node collected according to the product molding analysis through the UI.

[0076] The data enhancement unit 230 may include a space recognition module 231 , a workspace generation module 232 , an AR data enhancement module 233 , and a display information enhancement module 234 .

[0077] The space recognition module 231 converts an image of a real-world workspace input through a camera of the AR terminal 200 into a virtual 3D space.

[0078] The workspace generation module 232 generates a 3D matching coordinate system (x, y, z) based on at least one marker identified by the camera image.

[0079] The AR data enhancement module 233 tracks the position of the marker on the camera image as the user's line of sight (camera image focus) moves, and loads the AR data stored in the memory 145 into the real world in the camera image.

[0080] The AR data enhancement module 233 matches the augmented AR data in the real world with the position and posture (shape) of the product based on the 3D matching coordinate system (x, y, z).

[0081] The following describes in detail the problems of marker-based coordinate system generation in the prior art and describes a method for generating a 3D matching coordinate system based on space recognition using improved markers according to an embodiment of the present invention.

[0082] Figure 2 The process of identifying the position and posture of a test sample by a marker-based coordinate system in the prior art is shown.

[0083] See also Figure 2 , the position and posture of the product can be identified using the marker-based matching coordinate system in the prior art. The markers used to identify the position and posture of the real-world product are attached to the marker-based matching coordinate system in the prior art, so that the AR data can be matched in the product.

[0084] However, conventional techniques use one marker, or even if a plurality of markers are used, use a matching coordinate system based on point group matching.

[0085] Therefore, a mark attached to a product can only be identified when it is contained within the camera image (field of view), whereas if the mark is more than a certain distance away from the camera image and its area is reduced in the camera image, it cannot be identified, resulting in matching errors.

[0086] In addition, the more the AR terminal moves from the initial recognition point, the more the spatial tracking error accumulates, resulting in lower matching accuracy.

[0087] When generating AR data, the embodiments of the present invention improve matching accuracy by applying an improved product position and posture correction method based on spatial recognition that is different from the above-mentioned prior art.

[0088] For example, Figure 3 is a conceptual diagram of a method for correcting a position and a posture of a product based on space recognition according to an embodiment of the present invention.

[0089] Figure 3 1 shows a first camera image C#1 of a user wearing an AR terminal (HoloLens 2) before the user moves or before the user's line of sight moves according to an embodiment of the present invention. Figure 3 Also shown is a second camera image C#2 after the user or the user's line of sight has moved.

[0090] In an embodiment, in order to compensate for the problems of the marker-based matching coordinate system in the prior art, the AR data enhancement module 233 can use a reference marker M and multiple auxiliary markers M1 to M3 to apply a matching coordinate system correction based on spatial recognition in the marker-based 3D matching coordinate system, thereby improving the matching accuracy.

[0091] A matching coordinate system based on spatial recognition can be generated by three-dimensionally recognizing the real world with added auxiliary markers, and can be used to correct the position and posture of the product based on the system.

[0092] Whenever a marker is added to an image from a camera that moves with the user's line of sight, the AR data enhancement module 233 generates a marker-based matching coordinate system based on the added marker, and corrects the position and posture of the product by correcting the 3D matching coordinate system based on spatial recognition using the matching coordinate system.

[0093] Therefore, unlike the prior art, even if all markers are not in the image, the AR data enhancement module 233 can utilize the markers added by the identifier to correct the position and posture of the product through a matching coordinate system based on spatial recognition, thereby enhancing and matching the virtual AR data at the exact position of the product in the real world.

[0094] The display information enhancement module 234 may display information corresponding to the location information of each node of the product through the UI 235 , while enhancing the virtual AR data on the physical product.

[0095] The display information enhancement module 234 enhances the display information using a UI 235 that recognizes a user's hand based on a camera image. The UI 235 may include a user input menu for displaying at least one of an experiment, analysis, inspection, or evaluation of product formability.

[0096] Therefore, the display information enhancement module 234 can select a user input menu according to the information input through the UI 235 for identifying the user's hand, thereby displaying product trends, such as thickness reduction rate, panel sag rate and plastic strain rate of the AR data node matching the product.

[0097] Product trends can include the overall status of the product. For example, product trends can be displayed through color graphs.

[0098] The UI 235 includes a first UI 235-1 for identifying the left hand of the user to check a data list to be enhanced, and a second UI 235-2 for identifying the right hand of the user to select a desired data list and click a numerical value to be viewed.

[0099] UI 235 can enhance and display product trends through color graphs based on information input by recognizing the shape, movement, gesture, etc. of the user's left or right hand, such as the thickness reduction rate of the part corresponding to each node, the panel sag rate, the plastic strain rate, etc.

[0100] According to an embodiment of the present invention, the control part 240 controls the overall operation of the AR terminal 100 applied to stamping die products using at least one program and data for the program stored in the memory 241 .

[0101] The control unit 240 receives AR data processed by 3D data used by the stamping die department, enhances the processed data, improves the matching of the enhanced AR data with the physical product, and selectively displays the analysis results corresponding to the position information of each node through the UI, thereby maximizing the work efficiency of the stamping die department.

[0102] The control section 240 executes a program stored in the memory 241 to control the overall operation of components applied to the AR terminal 200 , and thus can operate as an actual control subject of the AR terminal 200 .

[0103] The control unit 240 may be implemented to include one or more processors operating according to a set program. According to an embodiment of the present invention, the set program may be programmed to perform the operation of the control method of the AR terminal applied to the stamping die process.

[0104] The control method of the AR system applied to the stamping die process according to the embodiment of the present invention is described in more detail below with reference to the accompanying drawings.

[0105] Figure 4 is a schematic flow chart of an AR system control method applied to a stamping die process according to an embodiment of the present invention.

[0106] See also Figure 4 , the control method of the AR system applied to the stamping die process according to the embodiment of the present invention is described based on the following assumptions, that is, the data generating unit 100: i) generates 3D data according to at least one of stamping die product design, 3D measurement or scanning, ii) generates display information corresponding to the position information of each node by analyzing the 3D data, and iii) transmits the 3D data and the display information to the AR terminal 200.

[0107] In operation S10 , the control part 240 of the AR terminal 200 receives 3D data corresponding to a stamping die product, which is input from the data generating part 100 , and generates information showing each node by analyzing the 3D data.

[0108] In operations S20 to S50 , the control part 240 converts the received 3D data into a form of a graphic mesh according to the type of form information of the 3D data, and matches the graphic mesh with the display information to generate AR data in a 3D virtual model form.

[0109] Operations S20 to S50 are described in more detail below.

[0110] In operation S20 , the control part 240 identifies the type of 3D data form information to determine whether the 3D data is in the form of a graphic mesh.

[0111] In operation S30, when the type of the form information of the 3D data is not in the form of a graphic mesh ("No" in operation S20), the control unit 240 converts the 3D data into a graphic mesh form of a polyhedral shape. For example, when the type of the 3D data is 3DCAD, the control unit 240 may convert the 3D data into the form of graphic mesh data.

[0112] When the type of the form information of the 3D data is the graphic mesh form ("YES" in operation S20), the control part 240 skips a separate data conversion process and proceeds to operation S40.

[0113] In operation S40 , the control part 240 lightweights the graphic mesh data by reducing the number of meshes of the graphic mesh data to be optimized for graphic processing.

[0114] In operation S50 , the control unit 240 generates AR data in the form of a 3D virtual model by matching the display information with the lightweight graphic mesh data.

[0115] In operations S60-S80, the control part 240 matches the AR data of the physical product displayed on the camera image and enhances the display information of each node, wherein the display information of each node is collected through the node 235 as a result of the product molding analysis.

[0116] Operations S60 to S80 are described in more detail below.

[0117] In operation S60 , the control part 240 converts a workspace of the real world input through the camera image into a virtual 3D space, and generates a 3D matching coordinate system based on a marker preformed in the workspace.

[0118] In operation S70, the control unit 240 tracks the position of at least one marker on the camera image when the user's line of sight moves, and loads the AR data into the real world to match the AR data with the position and posture of the product based on the 3D matching coordinate system. In an embodiment, whenever a marker is added to the image of the camera that moves with the movement of the user's line of sight, the control unit 240 can generate a matching coordinate system based on the added marker, and can correct the position and posture of the product by correcting the 3D matching coordinate system based on space recognition using the matching coordinate system.

[0119] In addition, the control unit 240 may improve the matching by enhancing the AR data based on the predefined coordinates of at least one inspection port of the product in actual mass production.

[0120] In operation S80 , the control part 240 enhances and displays display information corresponding to the position information of each node of the product through the UI 235 while matching with the AR data.

[0121] The control unit 240 can select a display menu based on information input through the UI 235 based on the recognition of the user's hand, and display product trends through color charts, such as the thickness reduction rate, panel sag rate, plastic strain rate, etc. corresponding to the position information of each node of the product.

[0122] For example, Figure 5 2 is a view showing an operation state of a user interface (UI) when CMM-based AR data is enhanced on a physical panel according to an embodiment of the present invention.

[0123] Figure 5A state is shown in which data result values ​​(gap, hole position, etc.) measured by a CMM are distinguished according to data types on a real-world physical panel (product) and enhanced on the first UI-1 235-1 for recognizing the left hand. In this case, the measured size result values ​​and models of parts connected to each other (not just one part) can be enhanced like the enhanced first UI 235-1, so that the relationship between the panel sizes can be understood.

[0124] Therefore, the manpower required to write past panel size trends and values ​​on the actual panel can be reduced, thereby improving work efficiency.

[0125] Figure 6 It shows a state where actual panel forming analysis results are enhanced in the form of a color map according to an embodiment of the present invention.

[0126] Figure 6 It is shown that according to an embodiment of the present invention, a panel trend is matched with an actual panel, and the panel trend is displayed via a color graph. The panel trend is a result of pre-analyzing the formability of the panel to confirm the formability of the panel.

[0127] According to an embodiment of the present invention, the thickness reduction rate, plastic strain rate, etc. obtained through the forming analysis of the panel can be expressed in the form of a color graph, so that the user can intuitively match and compare the analysis results with the actual panel, and use the second UI 235-2 to click his or her right hand to check the value of the target area.

[0128] Therefore, the problem in the prior art that after the panel is formed, the panel is inspected with the naked eye or the analysis result is printed in the form of a 2D image for comparison with the panel, and thus it is difficult to perform accurate and precise analysis can be solved.

[0129] Figure 7 It shows the UI state enhanced on the physical panel based on the panel scanning analysis result according to the embodiment of the present invention.

[0130] Figure 8 FIG. 4 shows a state where panel scan analysis results are enhanced in parallel on the inspection port according to an embodiment of the present invention.

[0131] Figure 7 and Figure 8 A state is shown in which a formed panel (product) is placed on an inspection port to inspect its state, and the analysis results obtained by measuring the gaps, holes, and surface states of the product by laser scanning are enhanced on the actual panel by a color map.

[0132] In the color map, the first color (e.g., green) portion indicates that the panel is in good condition compared to the drawing, the second color (e.g., blue) portion indicates that the panel is drooping downward relative to the inspection port, and the third color (e.g., red) portion indicates that the panel is higher than the inspection port.

[0133] As described above, the AR terminal 200 may enhance and display a color map representing a product status on a physical product to allow a more intuitive check of the overall status of the product.

[0134] The user may check a list of data to be enhanced through the first UI 235-1 using his or her left hand.

[0135] Furthermore, the user may select desired content through the second UI 235-1 and click a desired value to enlarge and check the value using his or her right hand.

[0136] Furthermore, the user can visually check the tendency of the panel through an enhanced color map matching the actual panel, and can check the measurement result value of the selected part by moving the camera image focus while controlling his or her sight.

[0137] AR data can be enhanced with scan analysis results obtained based on the inspection ports used in actual mass production to improve utilization and scalability, and data consistency can be improved by pre-defining coordinates to ensure a high degree of consistency.

[0138] As described above, according to an embodiment of the present invention, the 3D data used by the stamping die department can be converted into graphic mesh data, and the number of meshes can be reduced to achieve lightweight, thereby reducing the data processing burden and ensuring the responsiveness of AR data.

[0139] Unlike the existing technology, even if all markers are not in the image, the virtual AR data can be matched with the accurate position of the real-world product by identifying the added markers, and the position and posture of the product can be corrected using a matching coordinate system based on spatial recognition.

[0140] In addition, through the UI based on user hand recognition, the analysis information corresponding to the position information of each node of the selected product is enhanced and displayed, allowing users to intuitively check the analysis information of the required parts of the product at the work site.

[0141] The embodiments of the present invention may be implemented not only by the above-mentioned devices and / or methods. The embodiments of the present invention may be implemented by a program for executing functions corresponding to the configuration of the embodiments of the present invention, a recording medium storing the program, etc., and such implementation may be performed by a person of ordinary skill in the field to which the present invention relates based on the above-mentioned embodiments.

[0142] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto. It should be understood that various modifications and improvements made by ordinary technicians in this field using the basic concept of the present invention defined in the claims are included in the scope of the present invention.

[0143] <Description of Reference Numerals>

[0144] 100: Data Generation Department

[0145] 110: 3D data generation module

[0146] 120: Display information generation module 200: AR terminal

[0147] 110: Data input unit

[0148] 120: Data Processing Department

[0149] 221: Data conversion module

[0150] 222: Data Lightweight Module

[0151] 223: Display information matching module 230: Data enhancement unit

[0152] 231: Space recognition module

[0153] 232: Workspace generation module 233: AR data enhancement module 234: Display information enhancement module 235: UI

[0154] 235-1: First UI

[0155] 235-2: Second UI

[0156] 240: Control Department

[0157] 241: Memory

Claims

1. An augmented reality (AR) system for a stamping die, the AR system comprising: A data generating unit configured to generate three-dimensional 3D data corresponding to a stamping die product; and an AR terminal configured to convert the 3D data into AR data to be enhanced in the real world and match the AR data with a physical product, The AR terminal includes: A data input unit configured to receive the 3D data and display information corresponding to the position information of each node of the 3D data, a data processing unit configured to convert the 3D data into a polyhedral mesh according to the type of the form information of the 3D data, and generate AR data in the form of a 3D virtual model by matching with the display information, a data enhancement unit configured to enhance the AR data and match it with a physical product displayed on a camera image, and to enhance and display display information of each node of the stamping die product through a user interface UI, wherein the display information of each node is collected through a molding analysis of the physical product, and A control unit is configured to control the overall operation of components applied to the AR terminal.

2. The AR system according to claim 1, wherein: The data generation unit includes: a data generation module configured to generate the 3D data, the 3D data including at least one of 3D engineering data generated by modeling the stamping die product by a user, coordinate measuring machine data generated by measuring the state of the physical product, and 3D scanning data generated by scanning the physical product, and A display information generation module is configured to generate the display information by collecting at least one attribute information of the position of each node, hole processing status, surface, step, size, thickness and molding analysis information of the physical product according to the shape of the physical product.

3. The AR system according to claim 1, wherein: The data processing unit includes: a data conversion module configured to identify the type of form information of the 3D data and convert the 3D data into a graphic mesh form when the 3D data is not in a graphic mesh form, a data lightweight module configured to lightweight the graphic mesh by reducing the number of meshes input to the graphic mesh, and A display information matching module is configured to generate AR data to be enhanced in the form of a 3D virtual model in the real world by matching position information according to the shape of a lightweight graphic mesh with display information of each node.

4. The AR system according to claim 3, wherein: The data conversion module is configured as follows: Identifying the type of form information of the 3D data, When the 3D data is in the form of 3D computer-aided design (CAD), converting the 3D data into a graphical mesh form of a polyhedral structure, and When the 3D data is in the form of a graphics mesh, a separate data conversion process is skipped.

5. The AR system according to claim 1, wherein: The data enhancement unit comprises: a space recognition module configured to convert a real-world workspace inputted through the camera image into a virtual 3D space, a workspace generation module configured to generate a 3D matching coordinate system based on at least one marker identified by the camera image, an AR data enhancement module configured to load the AR data into the real world, and based on the 3D matching coordinate system, enhance the AR data and match the AR data with the position and posture of the product, and The display information enhancement module is configured to enhance the display information corresponding to each node position information of the product through the user interface UI in a state where the AR data is enhanced on the physical product.

6. The AR system according to claim 5, wherein: The AR data enhancement module is configured to: whenever a marker is added to an image of a camera that moves with the user's line of sight, Generate a matching coordinate system based on the added markers, and The position and posture of the product are corrected by correcting the 3D matching coordinate system based on space recognition using the matching coordinate system.

7. The AR system according to claim 5, wherein: The display information enhancement module is configured to enhance the display information based on the camera image by using a UI for recognizing a user's hand, wherein the UI includes a user input menu for displaying at least one of an experiment, analysis, inspection, and evaluation of product formability.

8. The AR system according to claim 7, wherein: The display information enhancement module is configured as follows: selecting the user input menu according to information input through the UI, and At least one product trend among a thickness reduction rate, a panel sag rate, and a plastic strain rate for each node corresponding to the AR data matched to the product is displayed by a color graph.

9. The AR system according to claim 5, wherein: The UI includes: a first UI configured to recognize a left hand of a user to check a list of data to be enhanced, and The second UI is configured to recognize the user's right hand to select a desired data list and click a value to be viewed.

10. A control method for an augmented reality (AR) terminal applied to a stamping die process, the control method comprising the following steps: Receiving three-dimensional 3D data corresponding to the input stamping die product and display information of each node of the stamping die product generated by analyzing the 3D data from the data generating unit; converting the 3D data into a graphic mesh form according to the type of the form information of the 3D data, and generating AR data in a 3D virtual model form by matching with the display information; matching the AR data with a physical product displayed via a camera image; and In a state of matching the AR data, display information corresponding to the position information of each node of the stamping die product is enhanced and displayed through a user interface UI.

11. The control method according to claim 10, wherein: Generating the AR data includes: identifying the type of form information of the 3D data to determine whether the 3D data is in a graphic mesh form, and When the type of the form information of the 3D data is not a graphic mesh form, the 3D data is converted into a graphic mesh form of a polyhedron shape.

12. The control method according to claim 11, wherein: Generating the AR data further includes: lightweighting the graphics mesh data for graphics processing by reducing the number of meshes of the graphics mesh data obtained by converting the 3D data, and The 3D virtual model is generated by matching lightweight graphic mesh data with the display information.

13. The control method according to claim 10, wherein: Matching the AR data includes: converting a real-world workspace inputted through the camera image into a virtual 3D space, and generating a 3D matching coordinate system based on pre-formed markers in the real-world workspace, and The AR data is matched with the position and posture of the product based on the 3D matching coordinate system by tracking the position of at least one marker on the camera image and loading the AR data into the real world.

14. The control method according to claim 13, wherein: The AR data matches the position and posture of the product including: Whenever a marker is added to an image of a camera that moves as the user's gaze moves, a matching coordinate system is generated based on the added marker; and The position and posture of the product are corrected by correcting the 3D matching coordinate system based on space recognition using the matching coordinate system.

15. The control method according to claim 13, wherein: Matching the AR data with the position and posture of the product includes: enhancing and matching the AR data based on predefined coordinates of at least one detection port of the physical product.

16. The control method according to claim 10, wherein: Enhancing and displaying said information includes: Based on the recognition of the user's hand, according to the information input through the UI, select to display a menu, and A product trend is displayed through a color graph, the product trend including at least one of a thickness reduction rate, a panel sag rate, and a plastic strain rate corresponding to position information of each node of the stamping die product.

17. The control method according to claim 10, wherein: Enhancing and displaying the display information includes: checking a list of data to be enhanced through a first UI for recognizing a left hand of a user, and Through the second UI for recognizing the user's right hand, a desired data list is selected and a numerical value to be viewed is clicked.

Images