Method for generating digital twin of augmented reality-based manufacturing facility and computing device for performing same

Abstract

A method for generating a digital twin for an augmented reality-based manufacturing facility and a computing device for performing the same are disclosed. A method for generating a digital twin according to one disclosed embodiment includes the steps of: generating a three-dimensional spatial model based on a three-dimensional spatial coordinate system inside a manufacturing facility based on captured images taken by a camera while moving inside the manufacturing facility and information obtained from an inertial measurement unit; detecting equipment inside the manufacturing facility based on the captured images and generating equipment-related information including equipment identification information and equipment alignment information for the equipment; obtaining a digital twin model of the equipment based on the equipment-related information; and placing the digital twin model in a three-dimensional spatial coordinate system based on the equipment-related information.

Description

Technology Field

[0001] The present invention relates to a digital twin generation technology for an augmented reality-based manufacturing facility. Background Technology

[0003] Recently, interest in digital twin technology has been increasing in the manufacturing industry for managing production facilities, workspaces, logistics flows, and maintenance status by reflecting them in a virtual space. Since digital twins can map the structure and equipment status of the actual manufacturing site to a virtual environment, they are recognized as one of the core technologies for smart factories and the digital transformation of manufacturing sites.

[0004] However, conventional methods for constructing digital twins for the manufacturing industry generally rely on manual 3D modeling of equipment and spaces or on precisely scanning the site using a separate 3D scanner to construct a virtual space based on the results. While this approach may offer the advantage of accurately reflecting the entire site, it requires expensive equipment and incurs significant time and cost for post-scanning processing and model alignment. In particular, maintaining the digital twin in a continuously up-to-date state is difficult in environments where manufacturing equipment is frequently added, modified, or moved.

[0005] Furthermore, conventional scanning-based technologies are primarily focused on reproducing spatial shapes or appearances exactly as they are, which limits their ability to structure and reflect information such as what kind of equipment the object is, what role the equipment performs, or management information corresponding to the equipment, such as operational information, performance information, and maintenance history. In other words, simply generating 3D data of the site's shape is insufficient to consistently support the object-unit identification, equipment registration, equipment information linkage, and operational utilization required in actual manufacturing sites. Prior art literature

[0007] Korean Patent Publication No. 10-2025-0105525 (July 8, 2025) The problem to be solved

[0008] An embodiment of the present invention is intended to provide a method for generating a digital twin of a manufacturing facility using a novel technique and a computing device for performing the same. means of solving the problem

[0010] A digital twin generation method according to one disclosed embodiment is a method performed on a computing device having one or more processors and a memory for storing one or more programs executed by said one or more processors, and comprises the steps of: generating a three-dimensional spatial model based on a three-dimensional spatial coordinate system inside a manufacturing facility based on information obtained from an inertial measurement unit and a captured image taken by a camera while moving inside the manufacturing facility; detecting equipment inside the manufacturing facility based on the captured image and generating equipment-related information including equipment identification information and equipment alignment information for said equipment; obtaining a digital twin model of said equipment based on said equipment-related information; and placing said digital twin model in said three-dimensional spatial coordinate system based on said equipment-related information.

[0011] The step of generating the above-mentioned facility-related information may include: a step of checking whether the detected facility is included in a pre-established facility data set; a step of calculating the recognition reliability of the facility detected in each frame of the captured image if the detected facility is included in the facility data set; and a step of determining whether to make the facility a target for creating a digital twin based on the calculated recognition reliability.

[0012] The digital twin generation method described above may further include the step of registering the facility as a candidate facility and managing the status of the candidate facility when the calculated recognition reliability is less than a preset threshold value.

[0013] The step of generating the above-mentioned facility-related information may include: determining the facility as a target for generating a digital twin when the calculated recognition reliability is greater than or equal to a preset threshold value; and extracting facility alignment information for placing the facility in the three-dimensional spatial model.

[0014] The step of extracting the equipment alignment information may include: a step of setting a bounding box corresponding to a three-dimensional spatial coordinate system for the equipment; and a step of extracting equipment alignment information including one or more of the center position, occupied area, placement direction, and rotation information of the equipment based on the bounding box.

[0015] The step of detecting equipment within the manufacturing facility may include: inputting the captured image into a pre-trained neural network model to detect equipment within the manufacturing facility, and classifying the detected equipment as unidentified equipment if the equipment is not included in the equipment data set; and further training the neural network model with the unidentified equipment as a new class.

[0016] For each facility, digital twin models are stored at multiple resolution levels, and the step of acquiring the digital twin models may involve selecting and acquiring one or more of the digital twin models at multiple resolution levels based on one or more of the processing performance of the computing device, memory usage, graphics processing performance, display mode, screen occupancy ratio of the facility, distance between the computing device and the facility, and network status.

[0017] For each facility, digital twin models are stored at multiple resolution levels, and the step of acquiring the digital twin models can involve selecting and acquiring digital twin models of different resolutions according to the importance level set for each facility.

[0018] The digital twin generation method described above may further include the step of classifying the detected equipment as unidentified equipment if the detected equipment is not included in a pre-established equipment data set; the step of creating a temporary object based on the shape information of the unidentified equipment; and the step of placing the temporary object in the three-dimensional spatial coordinate system.

[0019] The above digital twin generation method may further include the step of generating a management identifier for each digital twin model; and the step of transmitting a query including the management identifier to an external facility management system to receive operational management information of the facility corresponding to each digital twin model.

[0020] The step of receiving the above-mentioned operation management information may include: a step of searching for a plurality of facility record candidates corresponding to the digital twin model from the external facility management system using the above-mentioned management identifier; a step of calculating a matching score with the digital twin model for each of the searched plurality of facility record candidates; and a step of receiving the operation management information of the facility corresponding to the facility record candidate with the highest calculated matching score.

[0021] The above digital twin generation method may further include the step of generating a digital twin object by automatically mapping the operation management information of the facility to a digital twin model placed in the three-dimensional spatial coordinate system.

[0022] The above digital twin generation method may further include the step of generating relationship information including one or more of electrical connection relationships, physical connection relationships, and process flow relationships between each digital twin object; and the step of generating a digital twin graph for the digital twin objects of the manufacturing facility based on the relationship information.

[0023] The above digital twin graph may have the digital twin objects as nodes and relationship information between the digital twin objects as edges.

[0024] The step of generating the digital twin graph may include: a step of selecting a digital twin object located within a certain distance in the three-dimensional spatial coordinate system among a plurality of digital twin objects as a connection candidate equipment pair; and a step of performing a connection verification procedure based on one or more of the input and output directions of the connection candidate equipment pair, the relative distance of the connection candidate equipment pair, and whether the connection candidate equipment pair is a combination allowed by a predefined equipment connection rule.

[0025] The step of generating the digital twin graph may include: selecting a digital twin object located within a certain distance in the three-dimensional spatial coordinate system among a plurality of digital twin objects as a pair of candidate facilities for connection; calculating a connection score based on a plurality of pre-set verification elements for the pair of candidate facilities for connection; and determining an edge between the digital twin objects corresponding to the pair of candidate facilities for connection based on the calculated connection score.

[0026] A computing device according to one disclosed embodiment is a computing device comprising: a processor; and a memory for storing one or more programs executed by the processor, wherein the processor performs the operation of generating a three-dimensional spatial model based on a three-dimensional spatial coordinate system inside the manufacturing facility based on information obtained from a camera captured while moving inside the manufacturing facility and an inertial measurement unit; the operation of detecting equipment inside the manufacturing facility based on the captured image and generating equipment-related information including equipment identification information and equipment alignment information for the equipment; the operation of obtaining a digital twin model of the equipment based on the equipment-related information; and the operation of placing the digital twin model in the three-dimensional spatial coordinate system based on the equipment-related information. Effects of the invention

[0028] According to the disclosed embodiment, a three-dimensional spatial coordinate system of a manufacturing facility is generated using a camera and an inertial measurement unit, and detected equipment within the manufacturing facility is automatically placed in the three-dimensional spatial coordinate system as a predefined digital twin model or temporary object, thereby enabling the rapid and easy generation of a digital twin of a manufacturing site without relying heavily on expensive 3D scanners or separate manual modeling work as in the past.

[0029] In addition, by mapping the operational management information of the relevant equipment received from an external equipment management system to the deployed digital twin model, it becomes possible not only to simply visualize the digital twin model but also to link, display, and manage the operational management information of the equipment.

[0030] Furthermore, by generating relationship information between digital twin objects and creating a digital twin graph based on this, the graph structure reflecting the relationships between equipment can be utilized for production flow analysis, identification of bottlenecks, analysis of the impact of equipment replacement, and determination of maintenance priorities. Consequently, improvements in operational efficiency at the manufacturing site, precision in equipment management, and support for decision-making can be expected. Brief explanation of the drawing

[0032] FIG. 1 is a diagram showing the configuration of an augmented reality-based digital twin generation device according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating a method for generating a digital twin of a manufacturing facility according to one embodiment of the present invention. FIG. 3 is a flowchart illustrating the process of obtaining a digital twin model of a detected facility in one embodiment of the present invention. FIG. 4 is a flowchart illustrating a method for generating a digital twin graph of a manufacturing facility in one embodiment of the present invention. FIG. 5 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments. Specific details for implementing the invention

[0033] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0034] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the present invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0035] Additionally, terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0037] FIG. 1 is a diagram showing the configuration of an augmented reality-based digital twin generation device according to one embodiment of the present invention.

[0038] Referring to FIG. 1, the digital twin generation device (100) may include a space recognition module (102), an equipment recognition module (104), a digital twin construction module (106), and an equipment information linkage module (108).

[0039] The digital twin generating device (100) may be equipped with a camera and an inertial measurement unit (IMU). In one embodiment, the digital twin generating device (100) may be implemented in a smart device such as a smartphone or tablet, but is not limited thereto. In one embodiment, the digital twin generating device (100) may be a device for generating a digital twin of a manufacturing facility equipped with a plurality of facilities.

[0040] The spatial recognition module (102) can move inside the manufacturing facility where a digital twin is to be created and capture images with a camera. The spatial recognition module (102) can generate three-dimensional spatial information inside the manufacturing facility using captured images and information (acceleration information and angular velocity information) obtained from an inertial measurement unit. The spatial recognition module (102) can generate a three-dimensional spatial coordinate system based on Visual-Inertial SLAM (Simultaneous Localization And Mapping) based on captured images and information obtained from an inertial measurement unit.

[0041] In one embodiment, the spatial recognition module (102) may form a three-dimensional spatial coordinate system for the interior of the manufacturing facility by setting the first recognized floor surface or a reference location designated by a user within the manufacturing facility as the reference origin and setting an axis corresponding to the direction of gravity as the vertical axis. The three-dimensional spatial coordinate system may be used as a reference system to express the position, orientation, and arrangement relationship of the equipment object described later. The three-dimensional spatial coordinate system may be a reference coordinate system for implementing augmented reality.

[0042] The space recognition module (102) can generate a three-dimensional space model of the internal space of a manufacturing facility based on a three-dimensional spatial coordinate system. The three-dimensional space model may be a representation of the basic spaces of the internal space of the manufacturing facility, such as walls, floors, ceilings, columns, and passageways, as 3D space data. At this time, the space recognition module (102) can generate 3D space data in at least one form among a point cloud, a mesh, planar parameters, a depth map, and a surface map.

[0043] The equipment recognition module (104) can detect equipment within a manufacturing facility from a captured image taken by a camera. In one embodiment, the equipment recognition module (104) may be equipped with a neural network model that has been trained for object recognition. The equipment recognition module (104) can identify an area presumed to be equipment in the captured image and detect equipment by applying the neural network model to the identified area.

[0044] The equipment recognition module (104) can determine whether the detected equipment is included in a pre-established equipment data set. The pre-established equipment data set may store images, equipment model names, and equipment types of each piece of equipment in a matched manner. The equipment recognition module (104) can determine whether the detected equipment corresponds to any of the equipment included in the equipment data set. If the detected equipment corresponds to an equipment in the equipment data set, the equipment recognition module (104) can identify the equipment model name or equipment type of the equipment.

[0045] The equipment recognition module (104) can determine whether to make the equipment a target for creating a digital twin based on the recognition reliability of the equipment detected in each frame of the captured video when the detected equipment is included in a pre-built equipment data set. In one embodiment, the recognition reliability of the detected equipment may be calculated based on one or more of the classification score output by a neural network model, the continuity of positional changes between frames, the consistency of the equipment's external appearance, and image clarity. For example, the equipment recognition module (104) may determine the recognition reliability to be high when the classification score for the detected equipment is above a preset threshold value, when the positional changes between frames of the detected equipment are continuous, when the equipment's external appearance is consistent, or when the image clarity is high.

[0046] The equipment recognition module (104) can determine the equipment as a target for creating a digital twin if the calculated recognition reliability is greater than or equal to a preset threshold value. The equipment recognition module (104) can register the equipment as a candidate equipment if the calculated recognition reliability is less than a preset threshold value. The equipment recognition module (104) can manage the status of the candidate equipment. In one embodiment, the candidate equipment may be managed in any one of an initial detection state, a re-verification state, a confirmation waiting state, and a confirmation state. The candidate equipment may undergo a state transition depending on changes in recognition reliability, whether continuous detection is performed, and tracking stability. By managing the status of the candidate equipment step by step, temporary misrecognition due to changes in the field environment can be reduced and the reliability of digital twin creation can be improved.

[0047] When the equipment recognition module (104) determines the equipment to be the target for creating a digital twin, it can extract geometric alignment information for placing the equipment in a three-dimensional spatial model. For example, the equipment recognition module (104) can set a bounding box corresponding to a three-dimensional spatial coordinate system for the detected equipment, and based on the bounding box, extract equipment alignment information including one or more of the center position, occupied area, placement direction, and rotation information of the equipment.

[0048] The equipment recognition module (104) can classify the detected equipment as unidentified equipment if the equipment is not included in the equipment data set. The equipment recognition module (104) can extract shape information including size and external shape for the unidentified equipment. The shape information of the unidentified equipment can be used as additional training data to reinforce the equipment data set in the future. The equipment recognition module (104) can further train a neural network model by treating the unidentified equipment as a new class. Accordingly, equipment that was not initially a subject of recognition can be converted into identifiable equipment later through repeated shooting and training.

[0049] The digital twin construction module (106) can receive equipment-related information from the equipment recognition module (104). Here, the equipment-related information may include identification information of the equipment to be digital twin constructed (e.g., equipment model name, equipment type (class), etc.) and alignment information of the equipment (e.g., center location, occupied area, placement direction, and rotation information of the equipment). Based on the equipment-related information, the digital twin construction module (106) can obtain a digital twin model of the corresponding equipment.

[0050] The digital twin model may be predefined three-dimensional model data for the equipment. The digital twin model may include at least one of mesh data, surface data, texture data, equipment attribute information, and equipment operation parameters. At this time, the digital twin model may be called from the memory of the digital twin generating device (100) or received from a server outside the digital twin generating device (100).

[0051] In one embodiment, the digital twin construction module (106) may be configured to selectively call one or more of the digital twin models stored at multiple resolution levels for the same facility. For example, a low-resolution model, a medium-resolution model, and a high-resolution model may be stored in advance for a single facility. In this case, the digital twin construction module (106) may select one or more models based on the processing performance, memory usage, graphics processing performance, display mode (e.g., schematic display mode or detailed display mode, etc.), the screen occupancy ratio of the facility, the distance between the device and the facility, and network status, etc. of the device equipped with the digital twin generation device (100).

[0052] The digital twin construction module (106) can adaptively change the resolution of the digital twin model by monitoring the operating status of the device equipped with the digital twin generation device (100). For example, the digital twin construction module (106) can switch to a low-resolution model instead of a high-resolution model if the load increases above a preset standard based on one or more of the device's CPU usage, GPU usage, memory occupancy, rendering frame rate, and battery level. Conversely, if the device's computing resources are sufficient and stable rendering performance is secured, the low-resolution model can be replaced with a high-resolution model.

[0053] In one embodiment, the digital twin construction module (106) can select the resolution of the digital twin model based on the distance between the device and the equipment or the screen occupancy ratio. For example, if the equipment is located at a distance greater than a certain distance from the camera or if the area occupied on the screen is below a preset threshold, a low-resolution model can be selected. On the other hand, if the distance between the device and the equipment is less than a certain distance or if the equipment is displayed in an enlarged manner, a high-resolution model can be selected.

[0054] Additionally, the digital twin construction module (106) can use a mix of models of different resolutions according to the importance level set for each piece of equipment. For example, a high-resolution model can be selected and applied for equipment registered as a target of interest, equipment with a high probability of failure, equipment currently subject to maintenance, etc., while a low-resolution model can be selected and applied for auxiliary equipment or equipment required for simple location verification, etc.

[0055] The digital twin construction module (106) can calculate alignment parameters for placing the digital twin model in a three-dimensional spatial coordinate system. Specifically, the digital twin construction module (106) can calculate transformation values ​​as alignment parameters based on geometric alignment information transmitted from the equipment recognition module (104) so ​​that the digital twin model corresponds to the location and orientation where the actual equipment exists. At this time, the transformation values ​​may include translation transformation values, rotation transformation values, and scale transformation values.

[0056] The digital twin construction module (106) can align the reference point of the digital twin model based on the center position of the equipment. The digital twin construction module (106) can map the center coordinates of the equipment to the origin of the digital twin model and perform height direction correction so that the bottom contact surface of the equipment aligns with the bottom surface of the digital twin model. The digital twin construction module (106) can perform rotation alignment to reflect the orientation and attitude of the equipment. The digital twin construction module (106) can perform scale alignment (size enlargement or reduction) on the digital twin model by comparing the size of the equipment's bounding box and the volume of the occupied area with the reference size of the digital twin model.

[0057] The digital twin construction module (106) can perform additional corrections so that the digital twin model does not collide with the spatial structure. For example, by referring to the relationship with the wall surface, floor surface, ceiling, and the occupied area of ​​adjacent equipment in the 3D spatial model generated by the equipment recognition module (104), the position of the digital twin model can be finely adjusted so that it does not get inserted into the wall or sink below the floor and does not overlap with adjacent equipment.

[0058] The digital twin construction module (106) can create and place a temporary object corresponding to the equipment instead of calling a standard digital twin model when the equipment is unidentified equipment (i.e., equipment not included in the equipment data set). The digital twin construction module (106) can create a temporary object based on the shape information of the unidentified equipment. That is, the digital twin construction module (106) can operate in a dual structure in which it calls and places a pre-stored digital twin model for learned equipment, and creates and places a temporary object for unlearned equipment (i.e., unidentified equipment). The temporary object can be replaced with a digital twin model corresponding to the equipment if the equipment is registered as a new equipment class during the subsequent learning process.

[0059] The equipment information linkage module (108) can receive equipment-related information from the equipment recognition module (104). Additionally, the equipment information linkage module (108) can obtain information about a digital twin model placed on a three-dimensional spatial coordinate system. The equipment information linkage module (108) can structure each piece of equipment within the manufacturing facility into individual object units and create a digital twin object by combining semantic information and operational information with each object. That is, the equipment information linkage module (108) can convert each piece of equipment into an identifiable and manageable data object.

[0060] The equipment information linkage module (108) can generate a management identifier for each digital twin model. For example, the equipment information linkage module (108) can generate a management identifier by combining one or more of the model name of the equipment corresponding to the digital twin model, the type of equipment, the installation location of the equipment, and the arrangement order.

[0061] The equipment information linkage module (108) can perform information linkage with an external equipment management system using a management identifier as a key value. Here, the external equipment management system may be a manufacturing execution system (MES), an equipment asset management server, a maintenance management server, a production management server, a power management server, or a combination thereof. The equipment information linkage module (108) can transmit a query including a management identifier to the external equipment management system and receive operational management information of the equipment as a response. Here, the operational management information may include one or more of the following: specification information, performance information, equipment role information, power consumption information, operating status information, inspection cycle information, maintenance history, and failure history information of the equipment.

[0062] In one embodiment, the equipment information linkage module (108) can search for a plurality of equipment record candidates corresponding to the same model or the same type from an external equipment management system using a management identifier. The equipment information linkage module (108) can calculate a matching score for each of the searched plurality of equipment record candidates. Here, the matching score may be calculated based on at least one of the following: a distance error between the placement location of the digital twin model and the equipment location stored in the external equipment management system, a direction error between the placement direction of the digital twin model and the equipment direction stored in the external equipment management system, a degree of agreement with the equipment model name or equipment type, a degree of similarity with equipment size or shape information, a degree of agreement with the relative placement relationship with surrounding equipment, and a past shooting history or placement history. The equipment information linkage module (108) can select the equipment record candidate with the highest calculated matching score and receive operation management information of the corresponding equipment.

[0063] The equipment information linkage module (108) can generate a digital twin object by automatically mapping the operation management information of the equipment to a digital twin model placed in a three-dimensional spatial coordinate system. The digital twin object may be structured as an object unit by combining the digital twin model of the equipment within the manufacturing facility with the operation management information of the equipment.

[0064] At this time, the equipment information linkage module (108) can record the equipment name, manufacturer, model number, equipment role, installation date, rated power, current status value, alarm information, last inspection date and time, and maintenance personnel information in the attribute fields of the digital twin object. The equipment information linkage module (108) can connect sensor values, operating rates, production performance, fault codes, or alarm logs that are updated in real-time or periodically to the corresponding digital twin object to display the operating status of the equipment together. Through this, when a user or manager selects a specific piece of equipment in the digital twin space, they can simultaneously check not only the digital twin model of the equipment but also the operation management information and operating status of the equipment.

[0065] The equipment information linkage module (108) can generate relationship information between each digital twin object. For example, the equipment information linkage module (108) can generate relationship information regarding whether electrical connection relationships, physical connection relationships, or process flow relationships exist between equipment based on one or more of the relative distance between digital twin objects corresponding to each equipment, installation direction, line placement order, process rules, and predefined equipment connection rules.

[0066] The equipment information linkage module (108) can generate a digital twin graph of a manufacturing facility based on relationship information. For example, the equipment information linkage module (108) can represent each digital twin object as a node and relationship information between digital twin objects (e.g., electrical connection relationship, physical connection relationship, or process flow relationship) as edges in the form of a graph. Information such as the equipment's throughput, operating rate, waiting time, failure frequency, maintenance history, and current status value can be matched to each node. Information such as the amount of material moved between equipment, process sequence, signal transmission relationship, and power supply relationship can be matched to each edge. This graph structure can be utilized for production flow analysis, identification of bottlenecks, analysis of the impact of equipment replacement, or calculation of maintenance priorities.

[0067] The equipment information linkage module (108) may first select a pair of candidate equipment for connection in order to generate relationship information between a plurality of digital twin objects. For example, the equipment information linkage module (108) may select a first digital twin object and a second digital twin object located within a predetermined distance from each other in a three-dimensional spatial coordinate system as a pair of candidate equipment for connection. In addition to simple distance conditions, the equipment information linkage module (108) may set the priority of the candidate equipment for connection by referring to at least one of the installation direction of each equipment, the long axis direction of the equipment, the input / output direction, the line arrangement order, or whether a visible path exists in space.

[0068] The equipment information linkage module (108) can perform a connection verification procedure for candidate equipment pairs. For example, the equipment information linkage module (108) can determine whether the output direction of the first digital twin object and the input direction of the second digital twin object correspond to each other, and can determine whether the relative distance between the two digital twin objects or the expected connection path length is within a pre-set allowable range. Additionally, the equipment information linkage module (108) can determine whether the type of the first equipment and the type of the second equipment are combinations allowed under pre-defined equipment connection rules. For example, if a rule is defined so that only a specific inspection equipment or a transfer equipment can be connected after a specific processing equipment, the equipment information linkage module (108) can maintain the possibility of connection only for candidate equipment pairs that comply with the rule.

[0069] Additionally, the equipment information linkage module (108) can further verify candidate equipment pairs for connection using process sequence information, material flow information, production plan information, or equipment role information obtained from an external equipment management system. For example, the equipment information linkage module (108) may deny a connection between two equipment if, even if two spatially adjacent equipments exist, there is no sequential relationship in the process sequence information on the external system or no direct movement relationship in the material flow information. Conversely, even if physical distance or direction conditions are somewhat incomplete, if the process sequence information and equipment role information strongly match, the equipment information linkage module (108) may maintain the equipment pair as a connection target.

[0070] In one embodiment, the equipment information linkage module (108) can calculate a connection score for a pair of candidate equipment for connection based on a plurality of verification factors. Here, the connection score may be calculated based on at least one of relative distance suitability, direction matchability, compatibility between equipment types, process sequence matchability, material flow matchability, and arrangement relationship matchability with surrounding equipment. The equipment information linkage module (108) can confirm the edge between the first digital twin object and the second digital twin object only if the calculated connection score is greater than or equal to a preset threshold. Conversely, if the calculated connection score is less than the preset threshold, the relationship may be maintained in an unconfirmed state or configured to be re-verified after subsequent shooting or additional information collection.

[0071] Additionally, the equipment information linkage module (108) can determine the connection priority based on the connection score when there are multiple connection candidates for the same equipment object. For example, when one output equipment is placed adjacent to multiple subsequent equipment candidates, the equipment information linkage module (108) compares the connection scores for each candidate equipment and determines the equipment with the highest score as the primary connection target equipment, and may simultaneously confirm multiple connections as necessary. Accordingly, the connection relationship between equipment can be configured more rationally even in a complex manufacturing line.

[0072] The equipment information linkage module (108) can generate a digital twin graph based on the relationship information determined in this way. For example, each digital twin object is defined as a node, and relationship information that has passed the connection verification procedure can be defined as an edge. At this time, at least one of a physical connection relationship, an electrical connection relationship, a process flow relationship, a data communication relationship, or a power supply relationship may be assigned to each edge, and edge directionality, connection strength, or process priority information may be matched together as needed. Accordingly, a graph structure can be generated that reflects relationships meaningful to the actual manufacturing flow, not just information that the equipment is placed in close proximity.

[0073] Consequently, the equipment information linkage module (108) of the present invention can generate connection candidates based on the relative distance and direction between equipment, verify the connection candidates using equipment connection rules and process information from an external management system, and generate a digital twin graph of the manufacturing site by determining the edges based on the verification results. Accordingly, the present invention can provide an analyzable graph structure that goes beyond simple location-based adjacency relationships and reflects the actual process flow and equipment roles.

[0074] In one embodiment, the digital twin graph can be used to analyze the production flow of a manufacturing facility and identify bottlenecks. For example, the equipment information linkage module (108) can identify sections where processing delays are concentrated in the production flow by linking the throughput, operating rate, waiting time, failure frequency, downtime, or workload information of the equipment corresponding to each node. That is, if work is excessively concentrated in a specific node or a specific edge section on the digital twin graph, that section can be determined as a bottleneck. Accordingly, the manager can improve production efficiency by prioritizing the expansion of bottleneck equipment, adjusting the process order, or changing the layout structure between equipment.

[0075] In addition, digital twin graphs can also be used to analyze the impact of equipment replacement and determine maintenance priorities in manufacturing facilities. For example, if a specific piece of equipment is scheduled for failure, replacement, or inspection, the equipment information linkage module (108) can predict the impact of the equipment's interruption on the entire production line by tracking the preceding and succeeding equipment directly or indirectly connected to the equipment on the digital twin graph. Additionally, maintenance priorities can be determined regarding which equipment should be inspected or replaced first by synthesizing the importance, connection order, bottleneck contribution, failure history, and current status information of each piece of equipment.

[0076] According to the disclosed embodiment, a three-dimensional spatial coordinate system of a manufacturing facility is generated using a camera and an inertial measurement unit, and detected equipment within the manufacturing facility is automatically placed in the three-dimensional spatial coordinate system as a predefined digital twin model or temporary object, thereby enabling the rapid and easy generation of a digital twin of a manufacturing site without relying heavily on expensive 3D scanners or separate manual modeling work as in the past.

[0077] In addition, by mapping the operational management information of the relevant equipment received from an external equipment management system to the deployed digital twin model, it becomes possible not only to simply visualize the digital twin model but also to link, display, and manage the operational management information of the equipment.

[0078] Furthermore, by generating relationship information between digital twin objects and creating a digital twin graph based on this, the graph structure reflecting the relationships between equipment can be utilized for production flow analysis, identification of bottlenecks, analysis of the impact of equipment replacement, and determination of maintenance priorities. Consequently, improvements in operational efficiency at the manufacturing site, precision in equipment management, and support for decision-making can be expected.

[0079] In this specification, the term "module" may refer to a functional and structural combination of hardware for carrying out the technical concept of the present invention and software for driving said hardware. For example, the "module" may refer to a logical unit of a specific code and a hardware resource for executing said code, and does not necessarily refer to physically connected code or a single type of hardware.

[0081] FIG. 2 is a flowchart illustrating a method for generating a digital twin of a manufacturing facility according to an embodiment of the present invention. Although the method is described in the illustrated flowchart in a plurality of steps, at least some of the steps may be performed in a different order, combined with other steps, omitted, divided into detailed steps, or performed with one or more steps not illustrated added.

[0082] Referring to FIG. 2, the digital twin generating device (100) can generate a three-dimensional spatial model based on a three-dimensional spatial coordinate system inside the manufacturing facility based on captured images taken by a camera while moving inside the manufacturing facility and information obtained from an inertial measurement unit (S 101).

[0083] Next, the digital twin generation device (100) can detect equipment within a manufacturing facility based on captured images and generate equipment-related information including equipment identification information and equipment matching information for the equipment (S 103).

[0084] Next, the digital twin generation device (100) can obtain a digital twin model of the equipment based on equipment-related information (S 105). This will be described later with reference to FIG. 3.

[0085] Next, the digital twin generating device (100) can place the digital twin model in a three-dimensional spatial coordinate system based on equipment-related information (S 107).

[0087] FIG. 3 is a flowchart illustrating the process of obtaining a digital twin model of a detected facility in one embodiment of the present invention. In the disclosed flowchart, the method is described by dividing it into a plurality of steps, but at least some of the steps may be performed in a different order, combined with other steps and performed together, omitted, divided into detailed steps, or performed with one or more steps not shown added.

[0088] Referring to FIG. 3, the digital twin generating device (100) can check whether the detected equipment is included in a pre-built equipment data set (S 201).

[0089] If, as a result of the verification of S 201, the detected equipment is not included in the equipment data set, the digital twin generation device (100) can classify the equipment as unidentified equipment (S 203). Then, the digital twin generation device (100) can further train a neural network model with the unidentified equipment as a new class (S 205).

[0090] If, as a result of the verification of S 201, the detected equipment is included in the equipment data set, the digital twin generating device (100) can calculate the recognition reliability of the equipment detected in each frame of the captured image (S 207).

[0091] Next, the digital twin generating device (100) can check whether the calculated recognition reliability is greater than or equal to a preset threshold value (S 209).

[0092] If, as a result of the verification of S 209, the calculated recognition reliability is greater than or equal to a preset threshold value, the digital twin generation device (100) determines the equipment as the target for digital twin generation and can extract equipment alignment information for placement in a three-dimensional spatial model (S 211).

[0093] If, as a result of the verification of S 209, the calculated recognition reliability is less than a preset threshold value, the digital twin generation device (100) can register the equipment as a candidate equipment and manage the status of the candidate equipment (S 213).

[0095] FIG. 4 is a flowchart illustrating a method for generating a digital twin graph of a manufacturing facility in one embodiment of the present invention. In the disclosed flowchart, the method is described in a plurality of steps, but at least some of the steps may be performed in a different order, combined with other steps, omitted, divided into detailed steps, or performed with one or more steps not shown added.

[0096] Referring to FIG. 4, the digital twin generation device (100) can generate a management identifier for each digital twin model (S 301). Next, the digital twin generation device (100) can receive operation management information of the equipment corresponding to each digital twin model by transmitting a query including the management identifier to an external equipment management system (S 303).

[0097] Next, the digital twin generating device (100) can generate a digital twin object by automatically mapping the operation management information of the equipment to a digital twin model placed in a three-dimensional spatial coordinate system (S 305).

[0098] Next, the digital twin generating device (100) can generate relationship information including one or more of electrical connection relationships, physical connection relationships, and process flow relationships between each digital twin object (S 307).

[0099] Next, the digital twin generating device (100) can generate a digital twin graph by having each digital twin object as a node and relationship information between digital twin objects as edges (S 309).

[0101] FIG. 5 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in exemplary embodiments. In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0102] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a device for generating an augmented reality-based digital twin for a manufacturing facility. The computing device (12) may be a digital twin generating device (100).

[0103] The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) can cause the computing device (12) to operate according to the exemplary embodiment described above. For example, the processor (14) can execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0104] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0105] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0106] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0108] Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof. Explanation of the symbols

[0110] 10: Computing Environment 12: Computing device 14 : Processor 16: Computer-readable storage media 18: Communication bus 20 : Program 22 : Input / Output Interface 24 : Input / Output Devices 26: Network communication interface 100: Digital Twin Generator 102: Spatial Recognition Module 104: Equipment Recognition Module 106: Digital Twin Construction Module 108: Equipment Information Linkage Module

Claims

Claim 1 A method performed on a computing device having one or more processors and a memory storing one or more programs executed by said one or more processors, comprising: a step of generating a three-dimensional spatial model based on a three-dimensional spatial coordinate system within a manufacturing facility based on captured images taken by a camera while moving within the manufacturing facility and information obtained from an inertial measurement unit; a step of detecting equipment within the manufacturing facility based on the captured images and generating equipment-related information including equipment identification information and equipment alignment information for said equipment; a step of acquiring a digital twin model of said equipment based on said equipment-related information; and a step of placing said digital twin model in said three-dimensional spatial coordinate system based on said equipment-related information, wherein the step of detecting equipment within the manufacturing facility comprises inputting the captured images into a pre-trained neural network model to detect equipment within the manufacturing facility, and if the detected equipment is not included in a pre-established equipment data set, classifying said equipment as unidentified equipment; A digital twin generation method comprising the step of registering the unidentified facility as a new facility class based on a captured image of the unidentified facility and additionally training the neural network model to detect the new facility class. Claim 2 A digital twin generation method according to claim 1, wherein the step of generating equipment-related information comprises: a step of checking whether the detected equipment is included in a pre-established equipment data set; a step of calculating the recognition reliability of the detected equipment in each frame of the captured image if the detected equipment is included in the equipment data set; and a step of determining whether to make the equipment a target for generating a digital twin based on the calculated recognition reliability. Claim 3 A digital twin generation method according to claim 2, wherein the digital twin generation method further comprises the step of registering the facility as a candidate facility and managing the state of the candidate facility when the calculated recognition reliability is less than a preset threshold value. Claim 4 A method for generating a digital twin according to claim 2, wherein the step of generating equipment-related information comprises: determining the equipment as a target for generating a digital twin when the calculated recognition reliability is greater than or equal to a preset threshold value; and extracting equipment alignment information for placing the equipment in the three-dimensional spatial model. Claim 5 A digital twin generation method according to claim 4, wherein the step of extracting equipment alignment information comprises: a step of setting a bounding box corresponding to a three-dimensional spatial coordinate system for the equipment; and a step of extracting equipment alignment information including one or more of the center position, occupied area, placement direction, and rotation information of the equipment based on the bounding box. Claim 6 delete Claim 7 A digital twin generation method according to claim 1, wherein digital twin models at multiple resolution levels are stored for each facility, and the step of acquiring the digital twin model comprises selecting and acquiring one or more of the digital twin models at multiple resolution levels based on one or more of the processing performance of the computing device, memory usage, graphics processing performance, display mode, screen occupancy ratio of the facility, distance between the computing device and the facility, and network status. Claim 8 A digital twin generation method according to claim 1, wherein digital twin models are stored at multiple resolution levels for each facility, and the step of acquiring the digital twin models involves selecting and acquiring digital twin models of different resolutions according to a preset importance level for each facility. Claim 9 A digital twin generation method according to claim 1, wherein the digital twin generation method further comprises: a step of classifying the detected equipment as unidentified equipment when the detected equipment is not included in a pre-established equipment data set; a step of generating a temporary object based on shape information of the unidentified equipment; and a step of placing the temporary object in the three-dimensional spatial coordinate system. Claim 10 A digital twin generation method according to claim 1, wherein the digital twin generation method further comprises: a step of generating a management identifier for each digital twin model; and a step of transmitting a query including the management identifier to an external facility management system to receive operation management information of a facility corresponding to each digital twin model. Claim 11 A digital twin generation method according to claim 10, wherein the step of receiving the operation management information comprises: a step of searching for a plurality of facility record candidates corresponding to the digital twin model from the external facility management system using the management identifier; a step of calculating a matching score with the digital twin model for each of the searched plurality of facility record candidates; and a step of receiving the operation management information of the facility corresponding to the facility record candidate with the highest calculated matching score. Claim 12 A digital twin generation method according to claim 10, wherein the digital twin generation method further comprises the step of generating a digital twin object by automatically mapping operation management information of the facility to a digital twin model placed in the three-dimensional spatial coordinate system. Claim 13 A digital twin generation method according to claim 12, wherein the digital twin generation method further comprises: generating relationship information including one or more of electrical connection relationships, physical connection relationships, and process flow relationships between each digital twin object; and generating a digital twin graph for the digital twin objects of the manufacturing facility based on the relationship information. Claim 14 A digital twin generation method according to claim 13, wherein the digital twin graph has the digital twin objects as nodes and relationship information between the digital twin objects as edges. Claim 15 A method for generating a digital twin according to claim 14, wherein the step of generating the digital twin graph comprises: selecting a digital twin object located within a certain distance in the three-dimensional spatial coordinate system among a plurality of digital twin objects as a pair of candidate facilities for connection; and performing a connection verification procedure based on one or more of the input and output directions of the pair of candidate facilities for connection, the relative distance of the pair of candidate facilities for connection, and whether the pair of candidate facilities for connection is a combination allowed by a predefined facility connection rule. Claim 16 A method for generating a digital twin according to claim 14, wherein the step of generating the digital twin graph comprises: selecting a digital twin object located within a certain distance in the three-dimensional spatial coordinate system among a plurality of digital twin objects as a pair of candidate facilities for connection; calculating a connection score based on a plurality of pre-set verification elements for the pair of candidate facilities for connection; and determining an edge between the digital twin objects corresponding to the pair of candidate facilities for connection based on the calculated connection score. Claim 17 A computing device comprising: a processor; and a memory for storing one or more programs executed by the processor, wherein the processor performs the operation of generating a three-dimensional spatial model based on a three-dimensional spatial coordinate system within the manufacturing facility based on information obtained from an inertial measurement unit and a captured image taken by a camera while moving within the manufacturing facility; the operation of detecting equipment within the manufacturing facility based on the captured image and generating equipment-related information including equipment identification information and equipment alignment information for the equipment; the operation of acquiring a digital twin model of the equipment based on the equipment-related information; and the operation of placing the digital twin model in the three-dimensional spatial coordinate system based on the equipment-related information, wherein the operation of detecting equipment within the manufacturing facility includes the operation of inputting the captured image into a pre-trained neural network model to detect equipment within the manufacturing facility, and classifying the equipment as unidentified equipment if the detected equipment is not included in a pre-established equipment data set; and the operation of registering the unidentified equipment as a new equipment class based on the captured image of the unidentified equipment and additionally training the neural network model to detect the new equipment class. Claim 18 A computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises one or more instructions, and when the instructions are executed by a computing device having one or more processors, the computing device is configured to perform the steps of: generating a three-dimensional spatial model based on a three-dimensional spatial coordinate system within the manufacturing facility based on captured images taken by a camera while moving within the manufacturing facility and information obtained from an inertial measurement unit; detecting equipment within the manufacturing facility based on the captured images and generating equipment-related information including equipment identification information and equipment alignment information for said equipment; acquiring a digital twin model of said equipment based on said equipment-related information; and placing said digital twin model in the three-dimensional spatial coordinate system based on said equipment-related information, wherein the step of detecting equipment within the manufacturing facility comprises inputting the captured images into a pre-trained neural network model to detect equipment within the manufacturing facility, and if the detected equipment is not included in a pre-established equipment data set, classifying said equipment as unidentified equipment; A computer program stored on a non-transient computer-readable storage medium, comprising the steps of registering the unidentified facility as a new facility class based on a captured image of the unidentified facility and additionally training the neural network model to detect the new facility class.

Images