Extended reality-based on-site inspection support device and method

Abstract

The present invention relates to an augmented reality-based field inspection support device and method, wherein the augmented reality-based field inspection support device may include: a construction unit that constructs a 3D work site model by modeling the environment of a work site in 3D based on augmented reality; a providing unit that provides the 3D work site model to an augmented reality device of a worker; and a solution providing unit that provides a field inspection support solution to the augmented reality device to support field inspection of the work site in conjunction with the augmented reality device.

Description

Technology Field

[0001] The present invention relates to an extended reality (XR)-based field inspection support device and method. Background Technology

[0002] Korea's industrial accident rate is very high compared to OECD countries, and notably, it ranks first in industrial accident fatality, earning the dishonorable title of "workers' graveyard." The financial losses are substantial, with the direct and indirect costs incurred for a single fatal accident amounting to approximately 2 billion won.

[0003] On-site inspections of various equipment, machinery, and facilities within the site are crucial for preventing and avoiding accidents at industrial sites.

[0004] However, conventionally, on-site inspections (safety inspections) of industrial sites (work sites) were conducted by workers manually recording inspection results, which resulted in inaccuracies; there was a lack of an effective on-site inspection management system; inspections were inadequate due to a shortage of skilled personnel; inspection errors occurred because workers were not familiar with the information; and there was a lack of technology that allowed workers (on-site inspectors) to monitor the site conditions and inspection result information in real time.

[0005] Therefore, there is an urgent need for the development of technology that enables on-site inspections of industrial sites (work sites) to be performed more safely, efficiently, and conveniently.

[0006] The technology forming the background of the present invention is disclosed in Korean Published Patent Application No. 10-2004-0017642. The problem to be solved

[0007] The present invention aims to solve the problems of the aforementioned conventional technology and to provide an augmented reality-based field inspection support device and method that enables field inspections of industrial sites (work sites) to be performed more safely, efficiently, and conveniently.

[0008] The present invention aims to solve the problems of the aforementioned prior art and provides an augmented reality-based field inspection support device and method capable of resolving the issues associated with conventional field inspection methods (i.e., the fact that field inspections of industrial sites (work sites) are conducted by workers manually recording inspection results, which leads to inaccuracies; the absence of an effective field inspection management system; inadequate field inspections due to a shortage of skilled personnel; inspection errors caused by workers' lack of familiarity with information; and the absence of technology that allows workers (field inspectors) to monitor the site conditions and inspection result information in real time).

[0009] However, the technical problems that the embodiments of the present invention aim to solve are not limited to the technical problems described above, and other technical problems may exist. means of solving the problem

[0010] As a technical means for achieving the above-mentioned technical task, an augmented reality-based field inspection support device according to one embodiment of the present invention may include: a construction unit that constructs a 3D work site model by modeling the environment of a work site in 3D based on augmented reality; a providing unit that provides the 3D work site model to an augmented reality device owned by a worker; and a solution providing unit that provides a field inspection support solution to support a field inspection of the work site in conjunction with the augmented reality device to the augmented reality device owned by the worker.

[0011] In addition, the solution provider may provide an educational support service that enables safety education simulations to be performed through the safety education content by providing safety education content regarding the work site to the augmented reality device.

[0012] In addition, the solution provider may provide an inspection list management screen that allows checking and managing a checklist regarding inspection targets requiring inspection at the work site.

[0013] In addition, the solution provider may provide a 3D map screen that can be displayed on a 3D map corresponding to the 3D work site model by linking the information of the checklist with the 3D work site model.

[0014] In addition, when the solution providing unit detects that the operator has selected at least one of the inspection targets among the inspection targets in the information of the checklist as the current inspection target, it may generate guidance route information for the current inspection target by considering the GPS information of the augmented reality device and provide an inspection guidance screen displayed on the 3D work site model.

[0015] In addition, the solution providing unit may provide a remote support screen that displays an expert video captured from an expert terminal in a specific area of ​​the screen of the augmented reality device to enable communication with an expert when it is detected that a worker has clicked an expert support button displayed in a specific area of ​​the screen of the augmented reality device because expert support is needed during the progress of on-site inspection work at the work site.

[0016] In addition, the solution provider may provide a collaboration screen that enables a worker to proceed with on-site inspection work at the work site by sharing on-site inspection-related data with other workers and collaborating.

[0017] In addition, an augmented reality-based field inspection support device according to one embodiment of the present invention may further include a data management unit that stores inspection result data obtained from a worker through the augmented reality device in a database unit and derives an analysis result through analysis of the inspection result data.

[0018] In addition, the solution provider may provide a data analysis screen capable of visualizing the derived analysis results.

[0019] In addition, an augmented reality-based field inspection support device according to one embodiment of the present invention may further include a data sharing unit that transmits at least some of the data, including inspection result data obtained from a worker and analysis results, in real time to a manager terminal of a manager managing the work site or an evaluation institution terminal of an evaluation institution.

[0020] Meanwhile, an augmented reality-based field inspection support method by an augmented reality-based field inspection support device according to one embodiment of the present invention may include: a step of constructing a 3D work site model by modeling the environment of a work site in augmented reality-based 3D in a construction unit; a step of providing the 3D work site model to an augmented reality device owned by a worker in a providing unit; and a step of providing a field inspection support solution to support field inspection of the work site in conjunction with the augmented reality device in a solution providing unit to the augmented reality device owned by the worker.

[0021] The above-described means for solving the problem are merely exemplary and should not be interpreted as intended to limit the invention. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and the detailed description of the invention. Effects of the invention

[0022] According to the means for solving the problem of the present invention described above, by providing an augmented reality-based field inspection support device and method, it is possible to enable a worker to perform field inspections of an industrial site (work site) more safely, efficiently, and conveniently.

[0023] According to the means for solving the problem of the present invention described above, by providing an augmented reality-based field inspection support device and method, the conventional problem of inaccuracy in which field inspections of industrial sites (work sites) were conducted by having workers manually record inspection results can be resolved, enabling efficient data management through electronic documents, and providing an effective field inspection management system (service). Furthermore, the conventional problem of insufficient field inspection due to a shortage of skilled personnel can be resolved, enabling stable field inspection through remote collaboration with professional personnel (i.e., experts). Additionally, the conventional problem of inspection errors occurring due to workers' lack of knowledge of information can be resolved by providing real-time information and manuals to workers to prevent inspection errors. Moreover, through the provision of a real-time map service (i.e., a service providing a 3D map screen), workers (field inspectors) can monitor the situation at the site, inspection result information, etc., in real time, and the location of the workers can be shared with managers, etc.

[0024] However, the effects obtainable from the present invention are not limited to those described above, and other effects may exist. Brief explanation of the drawing

[0025] FIG. 1 is a diagram showing the schematic configuration of an augmented reality-based field inspection support system including an augmented reality-based field inspection support device according to one embodiment of the present invention. FIG. 2 is a schematic diagram showing the configuration of an augmented reality-based field inspection support system according to one embodiment of the present invention. FIGS. 3 to 10 are drawings illustrating examples of implementation of a service provided by an augmented reality-based field inspection support device according to an embodiment of the present invention. FIGS. 11 to 15 are drawings illustrating an example of a process in which a worker checks the status of inspection equipment using a 3D work site model provided by an augmented reality-based field inspection support device according to an embodiment of the present invention. FIG. 16 is a diagram illustrating a plurality of functions provided by an augmented reality-based field inspection support device according to one embodiment of the present invention. FIG. 17 is a diagram showing another example of the configuration of an augmented reality-based field inspection support system according to one embodiment of the present invention. FIG. 18 is a flowchart of an operation for an augmented reality-based field inspection support method according to an embodiment of the present invention. Specific details for implementing the invention

[0026] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0027] Throughout the specification of the present invention, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" or "indirectly connected" with other elements interposed between them.

[0028] Throughout the specification of the present invention, when a member is described as being located "on," "on the upper," "on the top," "under," "on the lower," or "on the bottom" of another member, this includes not only cases where the member is in contact with the other member, but also cases where another member exists between the two members.

[0029] Throughout the specification of the present invention, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0030] Throughout the specification of the present invention, some of the operations or functions described as being performed by a terminal, device, or device may instead be performed by a server connected to said terminal, device, or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal, device, or device connected to said server.

[0031] Throughout the specification of the present invention, the term "at least one" may be defined as a term including both singular and plural forms, and it will be obvious that even if the term "at least one" is not present, each component may exist in a singular or plural form and may mean singular or plural. Furthermore, whether each component is provided in a singular or plural form may be changed according to the embodiment.

[0032] FIG. 1 is a diagram showing the schematic configuration of an augmented reality-based field inspection support system (100) including an augmented reality-based field inspection support device (10) according to one embodiment of the present invention.

[0033] For convenience of explanation, the augmented reality-based field inspection support device (10) according to one embodiment of the present invention will be referred to as the device (10), and the augmented reality-based field inspection support system (100) according to one embodiment of the present invention will be referred to as the system (100). In addition, the details shown (described) in the drawings of the present invention, including FIG. 1, may be applied equally to the description of the device (10), even if they are omitted below.

[0034] Referring to FIG. 1, the system (100) may include the device (10), a worker-owned device (20), a manager terminal (30), and an expert terminal (40).

[0035] The present system (100) and the present device (10) are an extended reality (XR)-based field inspection support system and method, and may be referred to hereinafter by terms such as field inspection management system and method, field management system and method, etc.

[0036] The device (10) can provide remote field inspection support technology that enables efficient field inspection (safety inspection) of a work site (1) from a remote location.

[0037] The device (10) may be a device or server that provides at least one of a web page, app page, program, application (app), service, and platform related to augmented reality-based field inspection support. At this time, the program, application, service, and platform related to augmented reality-based field inspection support provided by the device (10) will be referred to as the program, the app, the service, and the platform, respectively, for convenience of explanation below. In the example illustrated in FIG. 1, the device (10) will be described as being provided in the form of a server capable of transmitting and receiving data via a network (5) with each of the worker-owned device (20), the administrator terminal (30), and the expert terminal (40). Accordingly, the device (10) may be referred to by other terms such as an augmented reality-based field inspection support server.

[0038] Users of the device (10) may include workers (20'), administrators (31), experts (41), etc. Users may use the service with or without installing the program or app using their own terminals. Additionally, users may use the service with or without registering (logging in) by accessing the program or app using their own terminals.

[0039] The worker-owned device (20) may refer to a device owned by the worker (20'). The worker (20') is a user who performs an on-site inspection (on-site inspection work, actual on-site inspection work) of the work site (1), and may be referred to by other terms such as inspector, on-site inspector, etc.

[0040] The work site (1) refers to a place where a worker (20') performs a site inspection, and is a place where an industry (work) actually takes place and proceeds, and may be referred to by other terms such as an industrial site. The work site (1) can be applied to sites in various fields, such as a manufacturing site. Here, sites in various fields may include sites where equipment inspection, safety inspection, site inspection, environmental hygiene inspection, maintenance, and training of personnel are carried out.

[0041] The worker-owned device (20) may include an augmented reality device (21) and a worker terminal (22). In describing the present invention, providing specific information (e.g., a field inspection support solution) to the worker-owned device (20) may mean providing specific information to at least one of the augmented reality device (21) and the worker terminal (22).

[0042] The extended reality device (21) may be a device that outputs various data (e.g., field inspection support solution, etc.) provided by the device (10) in 3D through extended reality (XR). The extended reality device (21) may be a device (wearable device) provided in a form that can be worn (mounted) on a part of the worker's (20') body, and may be, for example, Google Glass, a head-mounted display (HMD), or HoloLens 2 manufactured by Microsoft, which can provide an XR environment, but is not limited thereto, and any device capable of providing an XR environment may be applied. In the present invention, the extended reality device (21) may be referred to by other terms such as XR glasses or XR devices. The worker terminal (22) may refer to a terminal (personal terminal) possessed by the worker (20').

[0043] In the present invention, the extended reality device (21) is exemplified as an XR device capable of providing an extended reality (XR) environment, but is not limited thereto. The extended reality device (21) may be a device configured to provide at least one environment of reality among virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR). That is, the extended reality device (21) may provide at least one environment among a VR environment, an AR environment, an MR environment, and an XR environment.

[0044] The administrator terminal (30) may refer to a terminal possessed by an administrator (31) who manages and operates the device (10). The administrator (31) may be referred to by other terms, such as a site manager, a site supervisor, or a work supervisor who directs work to workers (20'), for example, as a user who manages a work site (1).

[0045] The expert terminal (40) may refer to a terminal possessed by an expert (41) that provides guide information for problem solving to the worker (20') through the device (10) when the worker (20') clicks the expert support button, when an unexpected situation or problem occurs while the worker (20') is conducting on-site inspection work. The expert (41) may be referred to by other terms such as professional personnel.

[0046] In FIG. 1, the configuration of the worker-owned device (20), the administrator terminal (30), and the expert terminal (40) is illustrated as having one of each provided within the system (100). However, this is merely an example to aid in understanding the invention and is not limited thereto. Multiple configurations may be provided, and the description of each configuration may be applied equally to the description of each of the multiple configurations, even if the details described below are omitted.

[0047] In the present invention, the terminal (i.e., worker terminal (22), manager terminal (30), and expert terminal (40) each) may include, for example, all types of wired and wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communication), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (WCode Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smartpad, tablet PC, laptop, wearable device, desktop PC, etc., but is not limited thereto.

[0048] The device (10) can transmit and receive data by being linked with each of the worker-owned device (20), the manager terminal (30), and the expert terminal (40) via a network (5), and can control the operation of each of the worker-owned device (20), the manager terminal (30), and the expert terminal (40).

[0049] The network (5) may include, for example, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a WIMAX (World Interoperability for Microwave Access) network, the Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a Bluetooth network, a NFC (Near Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) network, but is not limited thereto and may include various wired / wireless communication networks. A more detailed description of the device (10) is as follows.

[0050] Referring to FIG. 1, the device (10) may include a construction unit (11), a provision unit (12), a solution provision unit (13), a data management unit (14), a database unit (15), a data sharing unit (16), and a control unit (17).

[0051] The construction unit (11) can construct (create) a 3D work site model by modeling the environment (space) of the work site (1) in 3D based on extended reality (XR, eXtended Reality) (i.e., 3D based on an XR environment). That is, the 3D work site model can mean a model (data) that implements the shape (structure) of the work site (1) in a 3D form.

[0052] The providing unit (12) can provide (transmit) the 3D work site model built by the building unit (11) to the worker's (20') augmented reality device (21) via the network (5).

[0053] The solution provider (13) can provide a field inspection support solution to the augmented reality device (21) owned by the worker (20') to support field inspection (or field inspection work) of the work site (1) in conjunction with the augmented reality device (21). By providing the field inspection support solution to the augmented reality device (21), the solution provider (13) can ensure that the field inspection support solution is provided in a 3D form in the augmented reality environment (XR environment) provided by the augmented reality device (21). The field inspection support solution may be referred to by other terms such as solution data for field inspection support.

[0054] In the present invention, for example, when the solution provider (13) provides a field inspection support solution, it is exemplified that the field inspection support solution can be provided in a 3D form in an XR environment through an extended reality device (21). However, this is merely an example to aid in understanding the present invention and is not limited thereto. As another example, the solution provider (13) can provide the field inspection support solution to a worker terminal (22). That is, the solution provider (13) can provide the field inspection support solution in a 2D form through a worker terminal (22).

[0055] According to this, the data provided by the device (10) to the augmented reality device (21) in the present invention can be provided (displayed) in a 3D form in an XR environment, and the data provided by the device (10) to the worker terminal (22) can be provided (displayed) on the screen of the worker terminal (22) in a 3D form. As such, the augmented reality device (21) and the worker terminal (22) differ from each other only in that the display form of the data provided by the device (10) is different. Therefore, the description below regarding the data provided by the device (10) to the augmented reality device (21) can be applied equally to the description of the worker terminal (22), even if the content is omitted below. That is, the device (10) may provide the same data to the worker terminal (22) as the data provided to the augmented reality device (21) (e.g., data such as a field inspection support solution, an education support service, an inspection list management screen, a 3D map screen, etc.).

[0056] The solution provider (13) can provide an educational support service that enables safety education simulation to be performed through the safety education content by providing safety education content regarding the work site (1) to an augmented reality device (21) in relation to the above-mentioned field inspection support solution.

[0057] At this time, the administrator (31) may, for example, pre-register multiple safety education contents for providing safety education to workers in the database unit (15) through the administrator terminal (30). Each of these multiple safety education contents may refer to content (such as videos) created for the purpose of education regarding safety at the work site (1). Subsequently, the solution providing unit (13) may provide at least one safety education content selected by the worker (20') among the multiple safety education contents already registered in the database unit (15) to the augmented reality device (21). By providing the above safety education content to the augmented reality device (21), the solution providing unit (13) may enable the provision (display, presentation) of the safety education content to take place in an XR environment provided (implemented) by the augmented reality device (21).

[0058] In addition, the solution provider (13) may provide an inspection list management screen to an augmented reality device (21) that allows checking and managing a checklist (information of the checklist) regarding an inspection target requiring inspection at a work site (1) in relation to the above-mentioned field inspection support solution. Here, the inspection target may include equipment and a place. That is, the inspection target may be an inspection target equipment (inspection equipment) or an inspection target place (inspection place).

[0059] In addition, the solution provider (13) can provide a 3D map screen that can be displayed on a 3D map corresponding to the 3D work site model by linking the information of the checklist with the 3D work site model in relation to the above-mentioned field inspection support solution to an augmented reality device (21).

[0060] Additionally, the solution provider (13), in relation to the above-mentioned field inspection support solution, can provide an inspection guidance screen to the augmented reality device (21) by generating guidance route information for the current inspection target (or the screen of the augmented reality device (21) by considering the GPS information (i.e., current location information) of the augmented reality device (21) when it is detected that the worker (20') has selected at least one inspection target among the inspection targets in the information of the above-mentioned checklist as the current inspection target (location to move to) through the augmented reality device (21).

[0061] Here, the current inspection target refers to the inspection target that the worker (20') wishes to inspect while the worker (20') is conducting on-site inspection work (actual on-site inspection work) at the work site (20'), and may be referred to by other terms such as inspection target.

[0062] When the solution provider (13) detects that the worker (20') has selected a current inspection target, it obtains real-time location information of the augmented reality device (21) from a GPS sensor within the augmented reality device (21), then generates guidance route information that guides movement from the current location of the augmented reality device (21) to the location of the current inspection target selected by the worker (20'), and can provide an inspection guidance screen including the generated guidance route information to the augmented reality device (21). Here, the location of the current inspection target may refer to the location of the place that the worker (20') wishes to visit (move to), and may be referred to by other terms such as the place of

[0063] In addition, the solution provider (13) can provide a remote support screen to the augmented reality device (21) such that, in relation to the above-mentioned field inspection support solution, when it is detected that a worker (20') clicks an expert support button displayed in a specific area of ​​the screen of the augmented reality device (21) because expert support is needed during the field inspection work (actual field inspection work) at the work site (1), an expert captured video obtained from an expert terminal (40) is displayed in a specific area of ​​the screen of the augmented reality device (21) to enable communication (connection) with an expert (41).

[0064] Here, the expert video may refer to a video (image data) of an expert captured in real time from an expert terminal (40). The solution provider (13) can provide a remote support screen to an augmented reality device (21) so that when an unexpected situation or problem occurs while the worker (20') is conducting an on-site inspection, the worker can receive guide information for problem solving (i.e., overcoming an unexpected situation) from the expert (41) within the remote support screen and help the worker solve the problem in real time while communicating with the expert (41).

[0065] Additionally, when the worker (20') clicks the expert support button, the solution provider (13) may provide expert footage as previously mentioned, or may provide a pre-produced problem-solving guide video to the augmented reality device (21) for problem solving, thereby enabling the worker (20') to solve the problem based on the pre-produced problem-solving guide video. Here, the pre-produced problem-solving guide video may be, for example, a video that was previously produced by the administrator (31) and registered in advance in the database unit (15). Such a pre-produced problem-solving guide video may be referred to by other terms such as a pre-produced response manual video.

[0066] In addition, the solution provider (13) may provide a collaboration screen via an augmented reality device (21) that enables a worker (20') to collaborate with other workers by sharing field inspection-related data in relation to the above-mentioned field inspection support solution, thereby allowing field inspection work to proceed at the work site (1). Field inspection-related data may refer to various data generated / occurred in relation to field inspection at the work site (1).

[0067] Here, other workers may mean, for example, at least one of the remaining workers excluding one of the workers (20') who is one of the multiple workers using the device (10) (in particular, multiple workers performing on-site inspection of the work site (1)), ii) a manager (31), iii) an expert (41), and iv) at least one of the evaluation agency officials (not shown) of the evaluation agency terminal (not shown) included in the system (100) but not shown in the drawing.

[0068] In addition, the collaboration screen, for example, as a screen related to cooperation (co-work), may be referred to by other terms such as co-work screen.

[0069] The collaboration screen includes, for example, 1) a screen sharing menu that allows sharing the screen of the augmented reality device (21) or the screen of the worker terminal (22) with another worker's terminal; 2) a file sharing menu that allows sharing file data selected by the worker (20') on the augmented reality device (21) with another worker's terminal; 3) a recording menu (i.e., screen recording menu or video recording menu) that allows recording (shooting) and saving video acquired by the camera (image sensor) of the augmented reality device (21) or recording and saving a display video displayed on the screen of the augmented reality device (21); 4) a TTS (Text to Speech) menu that allows converting text data related to the on-site inspection of the work site (1) on the augmented reality device (21) into speech and providing it; 5) an AR drawing menu that enables the worker (20') to perform drawing work on the on-site inspection related data in an XR environment (or AR environment) and allows sharing the data drawn by the worker (20') (i.e., data drawn by the worker); and 6) the worker (20') and another worker It may include a video chat menu that enables video chatting.

[0070] As described above, the field inspection support solution provided by the solution provider (13) to the augmented reality device (21) may include an education support service, an inspection list management screen, a 3D map screen, an inspection guide screen, a remote support screen, a collaboration screen, a data analysis screen and a data management screen described later.

[0071] The data management unit (14) can store inspection result data obtained from a worker (20') through an augmented reality device (21) in a database unit (15) and derive an analysis result through analysis of the inspection result data. The data management unit (14) can provide the derived analysis result to the augmented reality device (21) or the worker terminal (22). The analysis result derived from the data management unit (14) may be referred to by other terms such as analysis result data.

[0072] The augmented reality device (21) can receive a field inspection support solution from the solution provider (13) and display it on the screen of the augmented reality device (21). The worker (20') can visit the work site (1) and perform a field inspection of the work site (1) (especially, actually perform the field inspection work). At this time, the worker (20') can perform the field inspection (field inspection work) while wearing the augmented reality device (21) on their body (e.g., face, head, etc.), and especially, by receiving the field inspection support solution provided by the solution provider (13) through the augmented reality device (21) worn by the worker (20'), the field inspection work can be performed based on the provided field inspection support solution.

[0073] When performing on-site inspection work, the worker (20') can perform on-site inspection work (i.e., perform actual inspection) on each of the inspection targets that require inspection among the inspection targets that can be inspected within the work site (1), and then use an augmented reality device (21) to input (create) inspection result data for each of the inspection targets for which on-site inspection work has been completed (i.e., inspection targets that have been inspected). Through this, the data management unit (14) can acquire the inspection result data(s) input by the worker (20') for each of the inspection targets for which on-site inspection work has been completed (i.e., inspection targets that have been inspected) from the augmented reality device (21), and store and manage them in the database unit (15).

[0074] In addition, the data management unit (14) can derive analysis results by performing, for example, statistical analysis on multiple inspection result data stored in the database unit (15). The data management unit (14) can derive and provide statistical analysis results by item (i.e., statistical analysis results by inspection completion target) as the analysis results by performing, for example, statistical analysis.

[0075] Additionally, the data management unit (14) can perform an analysis of the status of inspection completion and the status of reasons for non-completion of inspection based on a plurality of inspection result data stored in the database unit (15), for example, and through this, can derive analysis data on the status of inspection completion and analysis data on reasons for non-completion of inspection as analysis results and provide them to an augmented reality device (21) or a worker terminal (22).

[0076] Additionally, the data management unit (14) may provide a data management screen to an augmented reality device (21) that enables efficient management of multiple inspection result data as electronic documents in relation to multiple inspection result data stored in the database unit (15). At this time, by providing the data management screen, the data management unit (14) can control the worker (20') to perform at least one of the following functions on the inspection result data on the data management screen: a data inquiry function, a search function, an issuance function, a viewing function, an editing function, and a modification function.

[0077] In addition, the solution provider (13) may provide a data analysis screen capable of visualizing the analysis results derived from the data management unit (14) in relation to the field inspection support solution to an augmented reality device (21). For example, the data analysis screen may refer to a screen displaying data that visualizes the derived analysis results in real-time as a dashboard.

[0078] The data sharing unit (16) can transmit at least some of the data, including the inspection result data obtained from the worker (20') through the augmented reality device (21) and the analysis result (analysis result data) derived from the data management unit (14), in real time through the network (5) to the manager terminal (30) of the manager (31) who manages the work site (1), or to the evaluation agency terminal (not shown) of an evaluation agency official (not shown).

[0079] The data sharing unit (16) can transmit at least some of the data to enable the inspection result data or analysis result to be shared (information sharing) in real time with the manager (31) or an evaluation agency official (not shown), thereby allowing the manager (31) or an evaluation agency official (not shown) to check at least some of the data in real time remotely and take prompt action when an abnormality occurs, and can instruct (request) a specific task (specific task related to on-site inspection) to the worker (20') as needed.

[0080] The control unit (17) can control the operation of each part within the device (10). Additionally, the control unit (17) can control the operation (especially, screen display operation, etc.) of each of the components linked to the device (10) through the network (5) (e.g., an augmented reality device (21), a worker terminal (22), a manager terminal (30), a professional terminal (40), an evaluation agency terminal (not shown), etc.).

[0081] As described above, the device (10) can provide services (the service) and platforms (the platform) related to support for field inspections based on extended reality (XR). As the platform, the device (10) can provide a customized integrated field inspection solution platform using XR and mobile devices (i.e., worker terminals (22)). In providing the service (i.e., services related to support for field inspections based on extended reality), the device (10) can provide functions such as providing data sharing services, remote support and collaboration functions, and convenient configuration editing tools using XR and mobile devices.

[0082] By providing this service, the device (10) can enable a worker (20') to perform on-site inspections of the work site (1) more safely and to perform on-site inspection-related tasks efficiently. In addition, the device (10) can transform the site through linkage with XR technology and, by providing this service, can provide a convenient service that effectively prevents safety accidents occurring at the site in advance and enhances work efficiency. By providing this service, the device (10) can enable digital transformation of the industrial site (work site).

[0083] The device (10) can analyze problems that occur during the progress (performance) of on-site inspection by a worker (20') at a work site (1), and can store, evaluate, analyze, and manage all data generated in relation to the on-site inspection, and can provide visualized data through a dashboard, etc. The platform provided by the device (10) can be described as an XR on-site inspection platform specialized in strengthening on-site response capabilities.

[0084] The device (10) can provide, in relation to the service (a service related to support for field inspection based on augmented reality), a remote support function, a cross-platform function, a video recording function, a file sharing function, an AR drawing function, an AR pointing function, a data statistics and analysis function, a status monitoring function for monitoring the status related to field inspection, an XR device (i.e., an augmented reality device (21)) support function, a history management function for managing the history of field inspections, a location information inquiry function for looking up the location of a worker (20') or the location of an inspection target within the work site (1), a user (worker) registration function for registering a worker (20'), who is a user performing field inspections (field inspection work) at the work site (1), to the device (10), an inspection menu management function for managing field inspection menus, a data management function, a security management function, and a remote after-sales service support function.

[0085] By providing this service, the device (10) can provide an efficient field inspection management service to the worker (20'), enable efficient data management through electronic documents, enable stable field inspection (field inspection work) through remote collaboration with professional personnel (i.e., experts (41)), provide information related to the field inspection in real time or provide information such as manuals to the worker (20'), thereby preventing errors in the field inspection, and through a real-time map service (i.e., a service providing a 3D map screen), the worker (20') can monitor the situation of the work site (1), inspection result information, etc. in real time, and the worker's location can be shared with the manager, etc.

[0086] FIG. 2 is a schematic diagram showing the configuration of an augmented reality-based field inspection support system (100) according to one embodiment of the present invention.

[0087] Referring to FIG. 2, the device (10) can provide, for an administrator terminal (30), 1) a user registration function that allows an administrator (31) to register workers performing on-site inspections at a work site (1) in a database unit (15); 2) an inspection instruction function that allows an administrator (31) to instruct (request) an inspection task related to on-site inspection to a worker (20'); 3) an inspection status monitoring function that allows monitoring of the inspection status of on-site inspection work performed by a worker (20'); 4) a data management function that allows an administrator (31) to manage various data generated in relation to the work site (1); 5) a security management function that allows an administrator (31) to set and manage security for the managed data; and 6) an inspection history management function that allows managing the history of inspection result data obtained from workers.

[0088] In addition, the device (10) can provide, for example, a function for setting inspection equipment and zones, a function for checking the status of an inspection list, a function for managing inspection equipment and zones, a function for displaying inspection equipment and zones on a map, a function for identifying location information, a remote collaboration function through real-time screen sharing, a real-time data sharing function, a video recording function, a data storage function, an inspection result data analysis function, a function for automatically transmitting data to an administrator, and a function for providing notifications and re-notifications regarding the need for inspection measures.

[0089] In addition, the device (10) can provide the following functions to an augmented reality device (21) and a worker terminal (22) carried by a worker (20') who performs an inspection (on-site inspection) of a specific facility or a specific place within the work site (1).

[0090] For example, the device (10) may provide the inspection list management screen described above as an augmented reality device (21), wherein the inspection list management screen may provide, for example, 1) an inspection item editing function that allows the worker (20') to edit the item to be inspected (i.e., inspection item), 2) an inspection item status checking function that allows the worker (20') to check (monitor) the status of the inspection item, and 3) a user location information inquiry function that allows the worker (20') to look up location information. Accordingly, in the present invention, the term "inspection target" may be referred to by other terms such as inspection item, inspection target item, etc.

[0091] In addition, the device (10) can provide the above-described 3D map screen as an extended reality device (21), wherein the 3D map screen can provide, for example, 1) an inspection location information lookup function that can look up location information of an inspection target within a work site (1) (for example, looking up the location of an inspection target based on a 3D work site model), 2) an inspection information lookup function that can look up information of an inspection target, and 3) an inspection order registration function that can register, edit, and manage the inspection order of inspection targets.

[0092] In addition, the device (10) can provide the data analysis screen described above as an augmented reality device (21), wherein the data analysis screen can provide, for example, 1) a statistical analysis verification function for each item that can check the results of statistical analysis by inspection target (or inspection target that has completed inspection), 2) a dashboard visualization function that can visualize the analysis results as a dashboard, and 3) an inspection status monitoring function that can monitor the inspection status of inspection targets.

[0093] In addition, the device (10) can provide the aforementioned collaboration screen as an extended reality device (21), wherein the collaboration screen can provide, for example, 1) a screen sharing function through a screen sharing menu, 2) a file sharing function through a file sharing menu, 3) a screen recording function through a recording menu, 4) a TTS / STT service provision function through a TTS menu, 5) an AR drawing function through an AR drawing menu, 6) a video chat function through a video chat menu, etc.

[0094] FIGS. 3 to 10 are drawings illustrating an example of implementation of a service provided by an augmented reality-based field inspection support device (10) according to an embodiment of the present invention. In other words, FIGS. 3 to 10 are drawings illustrating examples of screens that can be displayed on the screen of a device (20) owned by a worker by the device (10). Herein, the device (20) owned by the worker may include at least one of an augmented reality device (21) and a worker terminal (22). For convenience of explanation, the following description will be based on the assumption that the device (20) owned by the worker is an augmented reality device (21).

[0095] Referring to FIGS. 3 to 10, for example, the control unit (17) of the device (10) can control the operation of the augmented reality device (21) so that screens such as those shown in FIGS. 3 to 10 are displayed on the screen of the augmented reality device (21) when it is detected that a worker (20') has accessed and logged in to the app or program through the augmented reality device (21).

[0096] Specifically, when the login is detected, the control unit (17) may display a basic screen as shown in FIG. 3 on the screen of the augmented reality device (21) as the first screen. Here, the basic screen may include a plurality of menu icons such as those indicated as 1 to 4, logo information of the app such as indicated as 5 (e.g., MDT Check), company logo information of a company operated by an administrator (31) such as indicated as 6 (e.g., Metadexter).

[0097] Additionally, the control unit (17) can display a checklist management screen, such as that shown in FIG. 4, on the screen of the augmented reality device (21). In the present invention, the term 'checklist' may be referred to as a checklist.

[0098] Here, the inspection list management screen comprises, for example, 1) an area displaying information about the currently active inspection list (i.e., information about the list of active inspection targets), as indicated by number 1; 2) a shortcut button to navigate to the detail page for each inspection target within the checklist, as indicated by number 2; 3) a button to navigate to the Management page related to inspection list management, as indicated by number 3; 4) an area displaying menu icons, as indicated by number 4; 5) an area to navigate to the detail page of an inspection target when a registered inspection target is clicked, as indicated by number 5; 6) an "Add Check Category" button, as indicated by number 6, which allows adding inspection targets (i.e., adding check items, adding items for inspection targets requiring inspection); 7) an on / off toggle button, as indicated by number 7, which allows setting the on / off status on the checklist for each registered inspection target; and 8) as indicated by number 8 It may include a detailed item setting area within a category where inspection items can be set in relation to the selected inspection target when a specific inspection target is selected among the registered inspection targets, 9) a detailed item editing button within a category where a list of inspection items related to the selected inspection target can be edited when the specific inspection target is selected as indicated in 9, 10) a detailed item adding button within a category where inspection items related to the selected inspection target can be added as indicated in 10.

[0099] FIG. 5 is a diagram showing another example of a checklist management screen. As another example, the control unit (17) can display a checklist management screen, such as the one shown in FIG. 5, on the screen of an augmented reality device (21).

[0100] At this time, referring to FIG. 5, the inspection list management screen may include an initial screen, an equipment registration screen, and an inspection item registration screen.

[0101] Here, the initial screen refers to the initial screen (first screen) of the inspection list management screen and may include: 1) a list viewing area where a list of saved inspection documents (i.e., a list of inspection targets) can be viewed as indicated by 1; 2) a settings icon menu where the equipment registration screen can be moved when the settings icon is clicked as indicated by 2; 3) an area where inspection status information is displayed for each registered inspection target as indicated by 3, allowing the inspection status information to be checked (wherein the inspection status information may include completed inspection status or incomplete inspection status); and 4) a shortcut button provided so that the operator can move to an inspection item registration screen where inspection items can be registered for each inspection target when clicked as indicated by 4.

[0102] The equipment registration screen may include: 1) a setting button that allows deleting and managing registered inspection targets (i.e., inspection equipment or inspection locations) as indicated by 1; 2) an on / off setting button that allows setting on / off for each registered inspection target as indicated by 2; and 3) an add button that allows registering additional inspection targets as indicated by 3.

[0103] The inspection item registration screen may include: 1) an edit button that can modify or delete the contents of an inspection item (an item of the inspection category) in relation to a specific inspection target as indicated by 1; 2) an add button that can add the contents of an inspection item as indicated by 2; and 3) a location display button that can provide map information of the inspection target stored in the device (10) as indicated by 3 (i.e., information showing the location of the inspection target on a 3D map in a 3D work site model).

[0104] Additionally, the control unit (17) can display a 3D map screen, such as that shown in FIG. 6, on the screen of the augmented reality device (21). The 3D map screen may refer to a screen that displays information (e.g., location information, history information, etc.) of inspection targets registered in an inspection list (checklist) on a 3D map corresponding to a 3D work site model.

[0105] Referring to (a) in FIG. 6, for example, the 3D map screen may include: 1) an area where the 3D map is displayed as indicated by 1, 2) an area where multiple menu icons are displayed as indicated by 2, and 3) location display information where the locations of inspection targets registered in the inspection list are displayed as indicated by 3.

[0106] Referring to (b) in FIG. 6, as another example, the 3D map screen may include (provide) information indicating the location of the inspection target (i.e., inspection equipment, etc.) as indicated by 1, information indicating the current location of the worker (20') performing the on-site inspection work as indicated by 2, information indicating the current location of the worker (20') performing the on-site inspection work as indicated by 2, 3) a sequence setting (setting) function that allows the worker (20') to set, specify, and change the order of performing the on-site inspection (i.e., the order of inspection) when performing the on-site inspection work through drag input or selection input for the inspection targets indicated on the 3D map as indicated by 3, and 4) a zoom in / out function that allows the 3D map displayed on the screen to be enlarged or reduced based on touch input with two fingers on the screen of the augmented reality device (21) or worker terminal (22) as indicated by 4.

[0107] Additionally, the control unit (17) can display a collaboration screen, such as the one shown in FIG. 7, on the screen of the augmented reality device (21). The collaboration screen may be referred to by other terms such as CoWork screen.

[0108] Referring to (a) in Fig. 7, for example, the collaboration screen may include 1) a screen sharing function button such as labeled 1, 2) a button displaying multiple menu icons such as labeled 2, and 3) an area where the screen (video) of the recipient receiving the shared content is displayed such as labeled 3.

[0109] In addition, referring to (b) in Fig. 7, as another example, the collaboration screen may include: 1) a file sharing button that enables file sharing as indicated by 1, 2) a TTS providing button that converts text into speech as indicated by 2, 3) an AR drawing providing button that provides drawing information as indicated by 3, and 4) a video recording button that provides a video recording function as indicated by 4.

[0110] In addition, the control unit (17) can display a data analysis screen, such as that shown in FIG. 8, on the screen of the augmented reality device (21).

[0111] Referring to (a) in FIG. 8, for example, the data analysis screen may include: 1) an analysis chart display area such as labeled 1, 2) multiple menu icon display buttons such as labeled 2, and 3) a button to change the alignment of the analysis chart such as labeled 3.

[0112] Referring to (b) in FIG. 8, as another example, the data analysis screen may include: 1) a "See More" button that allows checking analysis results such as the status of inspection completion or reasons for non-completion for inspection targets in the worker (20') inspection list, or analysis results of statistical analysis by item, as indicated by 1; and 2) a visualization area that provides real-time visualization of the analysis results on a dashboard as indicated by 2.

[0113] In addition, the control unit (17) can display (provide) an inspection guide screen, a remote support screen, a data management screen, a data analysis screen, etc., as shown in FIG. 9 on the screen of the augmented reality device (21).

[0114] At this time, the inspection guide screen may include (provide) a function to identify location information, which can identify the location of inspection targets among the inspection targets within the work site (1) (especially the location where the inspection targets are equipped in the 3D work site model or within the work site (1)); 2) a function to transmit location information using GIS; 3) a function to edit a convenient route or modify the location; 4) a function to provide guidance route information and a function to view and check the inspection list when the worker (20') selects a place to move to (i.e., current inspection target, desired place to move to); and 5) a function to provide a facility layout drawing of the work site (1) as a popup in one area of ​​the screen (i.e., a minimap in one area of ​​the screen) so that the worker (20') can easily identify the location of the current inspection target, which is the target to be inspected at present.

[0115] The remote support screen may include (provide) a function that provides expert footage captured by an expert terminal (40) to a specific area of ​​the screen of an augmented reality device (21) so that when a worker (20') encounters an unexpected situation or problem while conducting on-site inspection work, the worker can immediately collaborate with an expert (41) and respond to the problem by following the solution process, and a function that allows the worker (20') to check and record the configuration and status of the inspection target in real time. This remote support screen enables the worker (20') to check and record the configuration and status of the inspection target in real time, while enabling immediate communication with an expert (41) or manager (31), and allows even unskilled workers to be deployed immediately after receiving training on the spot, and helps to ensure that a response to the problem situation is made immediately remotely.

[0116] The data management screen may include (provide) functions that enable efficient data management through electronic documents, functions that allow a worker (20') to view, share, and issue data related to field inspections anytime and anywhere, and functions that allow data to be edited and modified on the spot. This data management screen may enable inspections to be conducted smoothly through an automatic verification function, allow inspection items to be checked conveniently, allow data to be edited and modified on the spot, and eliminate concerns about damage or loss of data.

[0117] The data analysis screen may include (provide) the following functions: 1) a function to collect and store various data related to on-site inspections; 2) a function to enable real-time storage, classification, and sharing of recorded data (i.e., data stored in the device (10); 3) a function to enable convenient data search and viewing; 4) a function to automatically aggregate inspection data and transmit inspection results in real-time to the manager (41) and the evaluation agency official; and 5) a function to provide a notification of necessary action or a request for re-inspection regarding the failed items to the augmented reality device (21) if there are failed parts among the inspection items. Additionally, the data analysis screen may provide the following functions: automatically aggregate and analyze data and then visualize it.

[0118] Additionally, the control unit (17) can control the operation of the providing unit (12) so that a 3D work site model constructed by the construction unit (11) as shown in FIG. 10 can be provided to an augmented reality device (21). The worker (20') can check and monitor the 3D work site model using the augmented reality device (21). The 3D work site model is data that 3D-enables an actual work site (1), and may refer to data in which actual inspection targets (inspection equipment) or the environment of the work site (1) are constructed in a 3D space.

[0119] The control unit (17) can enable the worker (20') to check (monitor) the status of actual equipment (inspection equipment to be inspected) in the work site (1) in real time through the 3D work site model. The control unit (17) can update the status of the inspection equipment in the 3D work site model in real time by reflecting the inspection result data for the completed inspection targets, which the worker (20') uploads (provides) to the device (10) by completing the field inspection work, into the 3D work site model in real time.

[0120] FIGS. 11 to 15 are drawings illustrating an example of a process in which a worker (20') checks the condition of inspection equipment using a 3D work site model provided by an augmented reality-based field inspection support device (10) according to an embodiment of the present invention.

[0121] Referring to FIGS. 11 to 15, the control unit (17) can update the status of inspection equipment within the 3D work site model in real time by reflecting in real time the inspection result data for the completed inspection targets uploaded (provided) to the device (10) by the worker (20') completing the field inspection work in the 3D work site model. The control unit (17) can control the operation of the providing unit (12) so that the 3D work site model updated in real time is provided to the worker (20') in real time through the augmented reality device (21), thereby enabling the worker (20') to check the current status of each inspection equipment through the 3D work site model.

[0122] Referring to FIG. 11, the control unit (17) can control the worker (20') to check the condition of the inspection target (e.g., inspection equipment) such as the switchboard circuit breaker through a 3D work site model. When the worker (20') checks the condition of the switchboard circuit breaker, the control unit (17) can enable the worker to check the condition of a plurality of inspection items, including 1) checking the condition of the switchboard and whether it is damaged, 2) checking the condition of the internal circuit breaker of the switchboard, 3) checking the condition and value of the pressure gauge, 4) checking for external leakage, and 5) checking the condition of piping and wiring.

[0123] Referring to FIG. 12, the control unit (17) can control the worker (20') to check the condition of the steam piping, for example, as an inspection target (e.g., inspection equipment) through a 3D work site model. When the worker (20') checks the condition of the steam piping, the control unit (17) can enable the worker to check the condition of a plurality of inspection items, including 1) checking the condition and numerical value of the pressure gauge (i.e., checking the pressure condition of the steam piping), 2) checking for non-water phenomena, 3) checking the temperature condition of the steam piping, 4) checking for steam leakage, and 5) checking the condition of the piping and steel structure.

[0124] Referring to FIG. 13, the control unit (17) can control the worker (20') to check the status of an inspection target (e.g., an inspection facility) through a 3D work site model. When the worker (20') checks the status of the integrated distribution panel, the control unit (17) can enable the worker to check the status of a plurality of inspection items, including: 1) checking the lighting status of the distribution panel (i.e., checking the operating status of the integrated distribution panel), 2) checking the position of the high-voltage circuit breaker operating lever, 3) checking for abnormalities in the wiring and terminals of the distribution panel, 4) checking for water leakage or dripping in the surroundings, and 5) checking for the attachment of warning or prohibition signs.

[0125] Referring to FIG. 14, the control unit (17) can control the worker (20') to check the condition of an inspection target (e.g., an inspection facility) such as a hot water tank through a 3D work site model. When the worker (20') checks the condition of the hot water tank, the control unit (17) can enable the worker to check the condition of a plurality of inspection items, including 1) checking the temperature of the hot water tank, 2) checking whether the circulation pump is operating, 3) checking whether the drainage tank is operating, 4) checking for leakage and water accumulation, and 5) checking for abnormalities in piping and wiring.

[0126] Referring to FIG. 15, the control unit (17) can control the worker (20') to check the condition of the fire extinguishing facility and gas pipe as an inspection target (e.g., inspection equipment) through a 3D work site model. When the worker (20') checks the condition of the fire extinguishing facility and gas pipe, the control unit (17) can enable the worker to check the condition of a plurality of inspection items, including 1) checking the internal and external condition of the fire hydrant, 2) checking the condition of the fire extinguisher, 3) checking the gas pipe cycle table, 4) checking for gas leakage, and 5) reading the meter inside the gas box.

[0127] FIG. 16 is a diagram illustrating a plurality of functions provided by an augmented reality-based field inspection support device (10) according to one embodiment of the present invention.

[0128] Referring to FIG. 16, the device (10) can provide the service including multiple functions as an extended reality device (21). Here, the multiple functions may include a real-time data sharing function, a communication function, a collaboration function, a remote support screen provision function (i.e., a remote support function), a configuration editing tool provision function, a location information identification function, a data management function for managing data as an electronic document, a 3D metaverse platform provision function, a damage care service provision function, etc.

[0129] Here, the device (10) can provide, in relation to the real-time data sharing function, real-time field recording and screen sharing functions via XR, and real-time storage functions for inspection list data via electronic documents. By providing the real-time data sharing function, the device (10) can enable the worker (20') to collaborate with experts, enable the manager to conduct objective inspections through real-time data sharing, and prevent errors (inspection errors) caused by artificial factors.

[0130] In addition, the device (10) can provide communication functions, such as a function to check and record product configuration and status in real time and enable immediate communication with a manager, and a function to convey additional inspection items or instructions through the provision of push notifications. By providing communication functions, the device (10) can enable smooth mutual communication between a worker and a manager, a worker and an expert, and a worker and other workers.

[0131] The device (10) can provide a function that enables data sharing and collaboration with experts in relation to collaboration functions, thereby enabling rapid work processing, enabling on-site training using XR, and enabling unskilled workers to be trained on the spot and deployed.

[0132] The device (10) provides a remote support screen provision function (i.e., a remote support function), thereby enabling the immediate sharing of an expert's solution motion in the event of an unexpected situation, and helping the worker (20', inspector) to solve the problem by following the guide information provided by the expert (41). In addition, regarding the remote support screen provision function, the device (10) can provide the worker (20') with countermeasures for the problem situation by recording an expert's response video (e.g., expert footage) and transmitting it to the augmented reality device (21) when communication is impossible or a problem situation occurs. Furthermore, when providing the remote support screen provision function, the device (10) can provide a response manual for the problem situation to the augmented reality device (21) by attaching it as a pop-up, for example, and thereby enable remote response to the problem situation.

[0133] The device (10) provides a configuration editing tool function, thereby enabling the worker (20') to conveniently edit and share configurations, and to smoothly conduct inspections (field inspections) through simple menu operations and modifications, and to conveniently share requested data and conveniently convey additional instructions to the worker (20').

[0134] The device (10) can provide a location information identification function, and in relation to the provision of the location information identification function, it can provide a location information transmission function using GIS, a function to view a guidance route and checklist when selecting a place to move, and a location guidance function using a facility layout popup.

[0135] The device (10) can provide a data management function that manages data as electronic documents, thereby enabling easy inspection through an automatic verification function, allowing the inspection to be completed according to the completion rate of the required process, allowing inspection items to be checked at any time in a pop-up menu, and effectively reducing (shortening) the data search time through a search function.

[0136] The device (10) can provide a 3D metaverse platform provision function, and when the 3D metaverse platform provision function is provided, when a worker (20') performs an inspection (i.e., performs an on-site inspection), the existing data is updated to the data after the inspection, and an avatar can move around in the digital twin or metaverse to check objects, and if a problem is discovered, a request for verification or a call can be made, and data information can be collected from each other through platform sharing between companies.

[0137] In addition, the device (10) can provide a damage care service function, and when providing the damage care service function, if a damaged part or replacement part is required during inspection, it can provide a function to order the desired item and quantity immediately by pressing the parts request button, a function to save clear data of the damaged or replacement-needed part by saving a photo (screenshot), a configuration in which a response team comes to repair or the data is delivered to an administrative staff member so that ordering can proceed on their behalf when the items are written and sent, and a function to always display the replacement-needed time as a popup.

[0138] In addition, the device (10) can provide a collection and storage function that can collect and store data related to on-site inspection of the work site (1), and can provide a function that automatically aggregates the collected data (i.e., inspection data, inspection result data) and transmits the inspection results in real time to the manager and the evaluation agency official, and a function that sends a notification of necessary action and a re-inspection notification in the case of parts that were insufficient among the inspection items.

[0139] Meanwhile, the service name of the service provided by the device (10) may be 'MDT_check' as an example, but is not limited thereto.

[0140] By providing this service, the device (10) can provide i) a function that supports on-site safety management using Virtual Reality, Augmented Reality, and Mixed Reality technologies, ii) a function that supports on-site safety training, on-site safety inspection, and on-site safety management.

[0141] By providing this service, this device (10) can provide the following effects (advantages): i) provide an environment similar to the actual site (actual work site (1)) to the worker (20') using XR technology, thereby maximizing the effectiveness of safety education; ii) support on-site safety management by monitoring safety conditions using real-time on-site information; and iii) provide real-time on-site information collection, analysis, and notification functions.

[0142] The system (100) may include an extended reality device (21) which is an XR device, solution software, safety education content, and the platform. The platform may be referred to as, for example, a safety inspection and management platform.

[0143] By providing this service, the device (10) can 1) provide on-site safety training to a worker (20') using XR technology in relation to on-site safety training, and increase the worker (20')'s understanding of safety by enabling them to perform simulations related to on-site safety in an XR environment through an extended reality device (21); 2) support on-site safety inspection using XR technology, and enable the worker (20') to simulate the site (work site (1)) and perform safety inspections using an extended reality device (21) which is an XR device; and 3) support on-site safety management using XR technology, and enable the worker to monitor the on-site safety situation and manage the safety situation using an XR device.

[0144] For example, the device (10) can analyze requirements (i.e., analyze requirements related to the equipment to be inspected) to define development goals for providing the service (or platform), perform equipment modeling (i.e., perform modeling to create a 3D model of the equipment to be inspected) so that it can be used in the system (100), develop (create) an algorithm to determine the status of the equipment, build and operate the system (100), and test the developed system (100) through testing and operation to verify whether it is operating normally.

[0145] By providing this service, the device (10) can monitor the condition of the equipment in real time, thereby extending the lifespan of the equipment and allowing for quick response in the event of equipment failure, which can reduce equipment maintenance costs. Additionally, by using XR technology, it can provide an environment similar to the actual site during equipment inspection, thereby increasing safety during inspection.

[0146] The device (10) can, for example, develop a Mobile XR Service operation management App and an XR Device solution to build the system (100), and in this regard, 1) develop an XR Contents-based field inspection solution operated in a Mobile XR App manner, 2) apply a function that allows an administrator (31) to add, register, group, and manage member workers, 3) provide a function that allows an administrator (31) to directly create and manage content through a Mobile App XR Editor, 4) apply a target registration and location setting function (including path) through an authoring tool installed in the App, 5) provide a target image setting function through the provision of image templates, and 6) apply statistical monitoring by data.

[0147] In addition, the system (100) can provide server hosting and can provide its own server management and provider cloud server hosting environments.

[0148] In addition, the system (100) and the device (10) can provide, in relation to the Mobile XR Service operation management App and XR Device management functions, 1) a function to support common categories between heterogeneous devices, 2) a function to create, modify, delete, and restore inspection content, 3) a function to monitor content, 4) a function to provide all rights of the user (administrator) and an integrated management system for XR content.

[0149] In addition, the present system (100) and the present device (10) may provide an XR editor function, and in this regard, may provide: 1) a function to apply rotation values ​​and position values ​​of the X, Y, and Z axes in real time through the editor, 2) a function to generate a QR code for each content, 3) a target image tracking function, 4) a terrain recognition function, etc.

[0150] In addition, the present system (100) and the present device (10) may provide a mobile app-based XR authoring tool. In this regard, the present system (100) and the present device (10) may provide a mobile app-based XR content creation support function, and in this regard, may provide: 1) an augmented reality authoring function through Unity, UWP, MRTK, NRSDK, and AR plugins; 2) a function that allows a user (e.g., a worker) to search, utilize, and upload desired models; 3) a function that supports FBX as a modeling data support file format; 4) an XR content preview function; and 5) a function that supports mobile browser environments such as Android / IOS. In addition, the present system (100) and the present device (10) may build a content creation and augmented reality creation user guide page, and may build a user guide page so that the user can easily use the content authoring tool.

[0151] At this time, the system (100) and the device (10) can provide support for mobile browser environments such as Android / IOS by using Flutter, for example. Flutter refers to a framework that allows Android and iOS development to be written as a single code. Flutter is written in the Dart language created by Google, and its form is similar to C, Java, or JavaScript. Therefore, there is no need to learn a separate language to handle Flutter. As you learn Flutter, you can naturally become familiar with the Dart language.

[0152] FIG. 17 is a diagram showing another example of the configuration of an augmented reality-based field inspection support system (100) according to one embodiment of the present invention. In particular, FIG. 17 is a diagram schematically showing the configuration of a system for providing the present app in the present system (100) as an example.

[0153] Referring to FIG. 17, the system (100) may include a client app that can be provided to, for example, a worker terminal (22) or an augmented reality device (21) of a worker (20') who is a client, the device (10) which is a server, a database, and an authentication server (not shown).

[0154] Here, the client app may refer to an app (the app) used by a user (e.g., a worker (20')) who uses the service provided by the device (10). This client app (the app) may provide functions to download, upload, modify, delete, etc., various contents (e.g., XR content, augmented reality content, etc.) provided by the device (10).

[0155] The device (10) may refer to a server that manages various contents provided by the device (10). The device (10) may, for example, provide functions for storing, managing, searching, classifying, etc. of the said contents. The device (10) may be constructed using various technologies, for example, the device (10) may be constructed using a web server, an application server, a database server, etc.

[0156] The database may be a configuration provided to enable the device (10) to store and manage the content. For example, the database may refer to a database unit (15) within the device (10), or it may be, for example, MySQL, MongoDB, Firebase Realtime Database, etc.

[0157] An authentication server (not shown) may be a server provided for the authentication of users (users) using the device (10). The device (10) may, for example, perform user authentication for users, and may perform user authentication using the authentication server. The device (10) may control user registration, login, and permission management for the device (10) for users who have completed user authentication using the authentication server (not shown).

[0158] The device (10) can perform software architecture design, interface design, data repository design, module design, user interface design, etc., to provide the service, for example.

[0159] Here, software architecture design may mean identifying what the subsystems or modules are that make up the entire system (100) and the relationships between the subsystems. Interface design may mean designing and defining interfaces between subsystems. Data storage design may mean designing files or databases, and module design may mean designing the algorithms of the modules that become components of the system (100), i.e., programs. Additionally, user interface design may mean designing menus, input forms, output screens, and printed materials.

[0160] The device (10) can, for example, derive and provide a final inference result in response to an API request (API Call) made from an augmented reality device (21) or a worker terminal (22). In this case, a RESTful API may be considered as the API in the system (100). A RESTful API refers to a web service that follows the Representational State Transfer (REST) ​​architecture style. REST uses the HTTP protocol, identifies resources by URLs, and manipulates resources using HTTP methods (GET, POST, PUT, DELETE, etc.). RESTful APIs generally process requests and responses using data in JSON or XML format.

[0161] Meanwhile, in the past, when conducting on-site inspections of industrial sites (work sites), there were various side effects (problems) when safety inspections (on-site inspections) were performed manually, such as: 1) difficulty in verifying whether the inspection was being carried out, 2) the inconvenience of having to do it manually, 3) difficulty in responding immediately when a problem occurred, 4) inability to share the location in real time, 5) the need for specialized personnel or prior training, 6) the possibility of replacing it with a mobile device but involving a limited screen and inconvenient operation process, and 7) a lack of vivid on-site realism.

[0162] To resolve these problems, the present invention provides the system (100) and the device (10) thereby enabling: 1) on-site inspection through real-time data sharing (allowing the recipient to perform objective inspection); 2) immediate data storage (recording of inspection screens and resolution of inspection lists through electronic documents); 3) remote virtual collaboration in case of a problem (including guide simulations and manuals); 4) the need for dispatching specialized personnel (allowing unskilled workers to perform the work by providing on-site training and guidance motions using XR); 5) high utility and time and cost savings; 6) smooth inspection through an expanded field of view and simple operation; and 7) the provision of optimal route and direction guidance services.

[0163] Below, we will briefly examine the operation flow of the present invention based on the details described above.

[0164] FIG. 18 is a flowchart of an operation for an augmented reality-based field inspection support method according to an embodiment of the present invention.

[0165] The augmented reality-based field inspection support method illustrated in FIG. 18 can be performed by the device (10) described above. Therefore, even if the content described below is omitted, the description of the device (10) can be equally applied to the description of the augmented reality-based field inspection support method.

[0166] Referring to FIG. 18, in step S11, the construction unit can construct a 3D work site model by modeling the environment of the work site in 3D based on extended reality (XR).

[0167] Next, in step S12, the providing unit can provide the 3D work site model to the worker's augmented reality device.

[0168] Next, in step S13, the solution provider may provide a field inspection support solution to support field inspection of the work site in conjunction with an augmented reality device, using the augmented reality device owned by the worker.

[0169] In the description above, steps S11 to S13 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Additionally, some steps may be omitted as necessary, and the order between steps may be changed.

[0170] An augmented reality-based field inspection support method according to one embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

[0171] In addition, the aforementioned augmented reality-based field inspection support method can also be implemented in the form of a computer program or application executed by a computer stored on a recording medium.

[0172] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0173] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention.

Claims

Claim 1 A construction unit that builds a 3D work site model by modeling the environment of the work site in 3D based on augmented reality; a providing unit that provides the 3D work site model to a worker's augmented reality device; and a solution providing unit that provides a field inspection support solution to the augmented reality device in conjunction with the augmented reality device to support field inspection of the work site. The system includes a data management unit that stores inspection result data acquired from a worker through the augmented reality device in a database unit and derives analysis results through analysis of the inspection result data; the solution providing unit provides an education support service that enables safety education simulations to be performed through the safety education content by providing safety education content regarding the work site to the augmented reality device; provides an inspection list management screen that allows checking and managing checklists regarding inspection targets requiring inspection at the work site; provides a 3D map screen that links the information of the checklist with the 3D work site model and displays it on a 3D map corresponding to the 3D work site model; and provides a collaboration screen that enables a worker to proceed with on-site inspection work at the work site by sharing on-site inspection-related data with other workers and collaborating; and the inspection list management screen includes an initial screen, an equipment registration screen, and an inspection item registration screen, wherein the initial screen includes a list viewing area for viewing a list of saved inspection targets, a settings icon menu that allows moving to the equipment registration screen when the settings icon is clicked, and an inspection status information displayed for each registered inspection target, such as "Inspection completed" or "Inspection status" It includes an area for checking inspection status information, including incomplete status, and a shortcut button provided to enable movement to an inspection item registration screen where inspection items can be registered for each inspection target when clicked by an operator, and the equipment registration screen includes a settings button that allows deleting and managing inspection targets, including registered inspection equipment or inspection locations.It includes an on / off setting button for setting on / off for each registered inspection target, and an add button for registering additional inspection targets; the inspection item registration screen includes an edit button for modifying or deleting the content of an inspection item, which is an item of the inspection category related to a specific inspection target, an add button for adding the content of an inspection item, and a location display button for providing map information of the inspection target, which is information displaying the location of the inspection target on a 3D work site model on a 3D map; the 3D map screen includes an area where the 3D map is displayed, an area where multiple menu icons are displayed, location display information displaying the locations of inspection targets registered in the inspection list, and worker's current location display information displaying the current location of a worker performing on-site inspection work on a 3D map; and includes an inspection location information lookup function for looking up location information of inspection targets within the work site based on the 3D work site model, an inspection information lookup function for looking up information of inspection targets, and an ability for a worker to register, edit, and perform on-site inspection order of inspection targets during on-site inspection work by drag input or selection input regarding inspection targets displayed on the 3D map. It is configured to provide a function for registering a manageable inspection sequence and a zoom in / out function that allows zooming in or out of a 3D map displayed on the screen based on two-finger touch input on the screen of an augmented reality device or a worker terminal, and the collaboration screen comprises: an area where the screen image of the recipient receiving the shared content is displayed; a screen sharing menu that allows sharing the screen of the augmented reality device or the screen of the worker terminal with another worker's terminal; a file sharing menu that allows sharing file data selected by the worker on the augmented reality device with another worker's terminal; and a recording menu that allows recording and saving video acquired by the camera of the augmented reality device or recording and saving video displayed on the screen of the augmented reality device.An augmented reality-based field inspection support device comprising: a Text-to-Speech (TTS) menu capable of converting text data related to field inspections at a work site into speech and providing it on an augmented reality device; an AR drawing menu capable of enabling a worker to perform drawing work on field inspection-related data in an XR environment and sharing data drawn by the worker; and a video chat menu capable of enabling a worker to perform video chat with other workers. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 An augmented reality-based field inspection support method by an augmented reality-based field inspection support device of claim 1, comprising: a step of constructing a 3D work site model by modeling the environment of a work site in augmented reality-based 3D in a construction unit; a step of providing the 3D work site model to a worker's augmented reality device in a providing unit; and a step of providing a field inspection support solution to support field inspection of the work site in conjunction with the augmented reality device in a solution providing unit to the augmented reality device owned by the worker.

Images