Extended reality-based guide visualization system for safety inspection of underground facilities

Abstract

The present invention relates to an extended reality-based guide visualization system for safety inspection of underground facilities. The extended reality-based guide visualization system for safety inspection of underground facilities provides a user with a video in which a virtual video is matched with a real video as the user wears a user device which is a wearable device, the system comprising: an information collection unit that collects external data including user information, underground facility modeling, location information, date information, and time information; a display unit that superimposes the external data onto a video captured from the user device and outputs same; a collaboration unit that performs collaboration by sharing and transmitting / receiving at least one piece of data from among real-time videos, voices, texts, images, 3D objects, models, and images between users by using wireless communication; a sensing unit that senses user environment information by providing a temperature measurement sensor, a harmful gas concentration measurement sensor, and a humidity measurement sensor at a preset position of the user device, and measures user health information by further attaching at least one medical sensor from among a thermal imaging camera, an electrocardiogram sensor, an oxygen saturation sensor, and a body temperature sensor; a risk level determination unit that determines a risk level of a corresponding user by using information collected by the sensor unit; and a monitoring unit that receives sensing data collected by the sensor unit and transmits an environmental state to a preset user terminal, wherein the sensor unit comprises: a sensor information collection unit that collects supply voltage supplied to the measurement sensor, consumption current, and output voltage; a sensor error measurement unit that extracts a comparison target time at which a measurement value change rate of the sensing data exceeds a preset ratio; and a data transmission unit that periodically transmits the comparison target time of the sensor error measurement unit to the monitoring unit, and if there is no time exceeding the preset ratio, the sensor error measurement unit extracts sensing data corresponding to n arbitrary times in the sensing data as the comparison target time.

Description

An augmented reality-based guide visualization system for safety inspection of underground facilities.

[0001] The present invention relates to a visualization system for underground facilities, and more specifically, to a visual inspection guide visualization system for underground facility safety inspection based on extended reality (XR) that visualizes underground facilities through a user terminal to which extended reality (XR) is applied and performs a safety inspection.

[0002] Industrial accidents, which refer to accidents or disasters that occur in factories, construction sites, mines, etc., can often result in loss of life and financial loss, and can lead to major disasters such as large-scale fires, explosions, structural collapses, and toxic substance leaks.

[0003] Additionally, these industrial accidents can lead to large-scale industrial accidents when they occur in response to the operation of large-scale facilities or equipment and the equipment used to support or operate them.

[0004] The causes of these industrial accidents vary depending on various factors, but generally, they can be caused by non-compliance with safety regulations, unsafe working conditions, defects in machinery and equipment, human factors (mistakes, carelessness, ignorance, etc.), and natural disasters. Among the causes of the aforementioned industrial accidents, the most solvable and practical factors are unsafe working conditions and human factors.

[0005] Meanwhile, in order to inspect, maintain, and repair large-scale facilities or underground facilities that are not visible from the outside in normal conditions, excavation work can be performed using data recorded at the time of construction.

[0006] However, there are often cases where the recorded data differs from the actual data due to various variables at the time of construction. In such cases, re-excavation is required, which incurs greater economic and time costs, and at the same time, the risk of damage to underground facilities and the probability of industrial accidents for workers may increase due to accidents that may occur during excavation.

[0007] Therefore, there is a need for the development of an extended reality-based guide visualization system for safety inspection of underground facilities that models underground facilities and displays underground facilities in a real environment on user terminals such as smart glasses and HMD (Head Mounted Display) using location-based data, thereby providing data on the location, status, excavation, maintenance, and repair history of underground facilities, enabling more efficient safety inspection of underground facilities.

[0008] The purpose of the present invention is to enable workers to check the location and information of underground facilities by modeling underground facilities and providing modeling data based on the location of a user terminal as an extended reality-based guide image.

[0009] Additionally, the purpose is to improve work efficiency and accuracy by sharing real-time images currently being collected using user terminals with experts and transmitting and receiving data.

[0010] Additionally, the purpose is to simplify worker health management and work communication by having workers wear user terminals that perform wireless communication, thereby collecting and sharing information such as worker location tracking, communication, connection with experts, current user health status, and environmental information status.

[0011] According to one embodiment of the present invention, an extended reality-based guide visualization system for safety inspection of underground facilities is provided, which provides an image in which a virtual image is aligned with a real image to a user by having the user wear a wearable device, and the system comprises: an information collection unit for collecting external data including user information, underground facility modeling, location information, date information, and time information; a display unit for superimposing the external data on an image captured from the user device and outputting it; a collaboration unit for performing collaboration by sharing and transmitting and receiving at least one or more data among real-time images, voices, texts, images, 3D objects, models, and images between users using wireless communication; a sensor unit for sensing user environment information by providing a temperature measurement sensor, a harmful gas concentration measurement sensor, and a humidity measurement sensor at a preset location of the user device, and for measuring user health information by further attaching at least one medical sensor among a thermal imaging camera, an electrocardiogram sensor, an oxygen saturation sensor, and a body temperature sensor; a risk determination unit for determining the risk of the corresponding user using information collected from the sensor unit; and a sensing unit for determining the risk of the user using the information collected from the sensor unit. A monitoring unit that receives data and transmits an environmental status to a preset user terminal, wherein the sensor unit includes a sensor information collection unit that collects a supply voltage, a consumption current, and an output voltage supplied to the measurement sensor, a sensor error measurement unit that extracts a comparison target time that is a time when a change rate of a measurement value of the sensing data exceeds a preset rate, and a data transmission unit that periodically transmits the comparison target time of the sensor error measurement unit to the monitoring unit, wherein the sensor error measurement unit can extract sensing data corresponding to n arbitrary times in the sensing data as comparison target times when there is no time exceeding the preset rate.

[0012] In addition, the display unit may include a modeling output unit that outputs the underground facility modeling corresponding to the user location information and the image being captured, a user recording unit that records preset inputs performed by the user wearing the user device in chronological order, an underground facility modeling update unit that reflects and outputs the modified underground facility modeling when the user wearing the user device modifies the displayed underground facility modeling, an underground facility notification unit that displays at least one notification among an underground facility inspection cycle, a parts replacement period, and a construction schedule date input by a previous user or a user sharing the server on the user device corresponding to preset conditions, and a matching unit that performs matching of the underground facility modeling output to the user device with a real-time captured image.

[0013] In addition, the risk determination unit may include a fever determination unit that determines whether there is fever in response to the user's body temperature measurement value collected by the sensor unit, an air quality determination unit that determines the atmospheric condition in response to the harmful gas concentration measurement value collected by the sensor unit, a fire determination unit that determines whether there is fire using a thermal imaging camera provided in the user device, and a health abnormality determination unit that determines whether there is an abnormality in the user's health condition when the electrocardiogram sensor and the oxygen saturation sensor are out of the standard range.

[0014] In addition, the sensor unit is provided with a comparison sensor as the same sensor within a preset radius from at least one of the temperature measurement sensor, the harmful gas concentration measurement sensor, and the humidity measurement sensor, and the monitoring unit uses the sensing data collected from the measurement sensor and the comparison sensor and the comparison target time to calculate the sensor failure probability (S) according to the following [Mathematical Formula 1] TR ) is produced,

[0015] [Mathematical Formula 1]

[0016]

[0017] (Here, S TR is the probability of sensor failure, T ms1 is the measurement value of the measurement sensor at the first time, T ms2 is the measurement value of the measurement sensor at the second time, T ms3 is the measurement value of the measurement sensor at the third time, T msn is the measurement value of the measurement sensor at the nth time, T cst1 is the measurement value of the comparison sensor at the first time, T cst2 is the measurement value of the comparison sensor at the second time, T cs3 is the measurement value of the comparison sensor at the third time, T csn is the measurement value of the comparison sensor at the nth time, T msav is the average measurement value of the measurement sensor, and SE means the efficiency deviation of the measurement sensor.)

[0018] The efficiency deviation (SE) of the above measurement sensor is calculated according to the following [Mathematical Formula 2],

[0019] [Equation 2]

[0020]

[0021] (Here, SE av is the average efficiency of the measuring sensor, SE i refers to the initial efficiency of the measuring sensor)

[0022] The average efficiency of the above measurement sensor can be calculated according to the following [Mathematical Formula 3].

[0023] [Equation 3]

[0024]

[0025] (Here, W IN1 is the power supplied to the measurement sensor at the first time, W OUT1 The power measured when the measurement value sensed by the measurement sensor at the first time is output as an electrical signal, W IN2 is the power supplied to the measurement sensor at the second time, W OUT2 The power measured when the measurement value sensed by the measurement sensor at the second time is output as an electrical signal, W INnis the power supplied to the measurement sensor at the nth time, W OUTn It means the power measured when the measurement value sensed by the measurement sensor at the nth time is output as an electrical signal.

[0026] In addition, the monitoring unit further reflects the weight according to the distance of the measurement sensor and the comparison sensor, and calculates the sensor failure probability (S) according to [Mathematical Formula 4] and [Table 1] below. TR ) can be produced.

[0027] [Equation 4]

[0028]

[0029] (Here, D v refers to the distance weighting of the measurement sensor and the comparison sensor)

[0030] [Table 1]

[0031]

[0032] According to the present invention, by modeling underground facilities and providing modeling data based on the location of a user terminal as an extended reality-based guide image, a worker can check the location and information of underground facilities.

[0033] Additionally, by sharing real-time images currently being collected using user terminals with experts and transmitting and receiving data, work efficiency and accuracy can be improved.

[0034] Additionally, by having all workers wear user terminals that perform wireless communication, it is possible to identify the worker's location, communicate, connect with experts, and collect and share information such as the current user's health status and environmental information, simplifying worker health management and work communication.

[0035] FIG. 1 is a block diagram of an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention.

[0036] FIG. 2 is a drawing showing an intermediate block diagram of a display unit in an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention.

[0037] FIG. 3 is a drawing showing an intermediate block diagram of a sensor unit in an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention.

[0038] FIG. 4 is a drawing showing an intermediate block diagram of a risk assessment unit in an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention.

[0039] FIG. 5 is a drawing showing a visualized image of an underground facility through an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention.

[0040] FIG. 6 is a drawing illustrating an example of a time-lapse through an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention.

[0041] FIG. 7 is a drawing illustrating another example of a time-lapse through an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention.

[0042] FIG. 8 is a drawing illustrating an example of spatial information alignment using an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention.

[0043] Specific details, including the problems to be solved, means of solving them, and effects of the invention, are included in the embodiments and drawings described below. The advantages and features of the present invention, and methods for achieving them, will become clearer with reference to the embodiments described below in detail, along with the accompanying drawings.

[0044] The scope of the present invention is not limited to the embodiments described below, and various modifications may be made by a person having ordinary knowledge in the relevant technical field without departing from the technical spirit of the present invention.

[0045] Hereinafter, the extended reality-based guide visualization system for safety inspection of underground facilities according to the present invention is described in detail with reference to the attached drawings 1 to 8.

[0046] First, FIG. 1 is a block diagram of an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention, FIG. 2 is a diagram showing an intermediate block diagram of a display unit in an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention, FIG. 3 is a diagram showing an intermediate block diagram of a sensor unit in an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention, FIG. 4 is a diagram showing an intermediate block diagram of a risk assessment unit in an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention, FIG. 5 is a diagram showing a visualized image of an underground facility through an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention, FIG. 6 is a diagram showing an example of a time-lapse through an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention, and FIG. 7 is a diagram showing an example of a time-lapse through an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention. This is a drawing showing another example of a time-lapse using an extended reality-based guide visualization system, and FIG. 8 is a drawing showing an example of spatial information alignment using an extended reality-based guide visualization system for safety inspection of underground facilities according to an embodiment of the present invention.

[0047] Referring to FIG. 1, an extended reality-based guide visualization system (100) for safety inspection of underground facilities according to an embodiment of the present invention may include an information collection unit (110), a display unit (120), a collaboration unit (130), a sensor unit (140), a risk assessment unit (150), and a monitoring unit (160).

[0048] The extended reality-based guide visualization system (100) for safety inspection of underground facilities of the present invention can provide a user with an image in which a virtual image is aligned with a real image by having the user wear a wearable device, such as a user device.

[0049] Here, the user device means a device that allows the user to experience virtual reality (VR) or augmented reality (AR) as if in a 3D space by projecting a 3D image using parallax through a small display located close to both eyes of the user, a gyro sensor that tracks the user's movements, a rendering function that creates an image according to the movement, etc., and examples thereof include head mounted displays (HMDs) and smart glasses that are currently in popular use.

[0050] The above information collection unit (110) can collect external data including user information, underground facility modeling, location information, date information, and time information from a preset user terminal.

[0051] Here, the preset user terminal may refer to a PC, smartphone, laptop, tablet PC, HMD, smart glasses, etc. capable of transmitting and receiving data using wired or wireless communication.

[0052] At this time, data transmission and reception using the preset user terminal can be continuously performed in response to the time selected by the user and the cycle set by the user.

[0053] Meanwhile, the user information may refer to personal information of the user wearing the user device, such as the user's name, age, employee number, gender, past medical history, and current illness, and the personal information collected may be increased or decreased depending on the user's role, such as worker, manager, or supervisor.

[0054] In addition, the underground facility modeling can be performed by creating a model using a game engine such as Unity, Unreal Engine, etc., designing the underground facility modeling with the shape, texture, animation, etc. of the object using 3D design software, and then transmitting it to the user device so that it can be output in conjunction with sensors, cameras, touch controllers, etc. within the user device.

[0055] Additionally, the above location information may refer to information collected from a Geographic Information System (GIS), which is a computer-based system that collects, stores, manages, analyzes, and visualizes geographical data.

[0056] Through this, the geographic data visualized on the user device can be expressed as a map, graph, chart, 3D model, etc.

[0057] The above display unit (120) can output external data by overlaying it on an image captured from the user device.

[0058] Here, the display unit (120) is described in more detail with reference to FIG. 2.

[0059] Referring to FIG. 2, the display unit (120) may include a modeling output unit (121), a user record unit (122), an underground facility modeling update unit (123), and an underground facility notification unit (124).

[0060] The above modeling output unit (121) can output underground facility modeling corresponding to user location information collected through a user device and the image being captured.

[0061] Here, the modeling output unit (121) performs object detection, tracking, and key point extraction from the underground facility modeling and location information collected by the information collection unit (110) and real-time images captured by the user device to identify objects (buildings, signs, roads, signboards, etc.) having a specific pattern, and renders and synthesizes the underground facility modeling based on the object into a real-time image.

[0062] At this time, the synthesized underground facility model can be projected onto an image, and visual effects such as lighting, shadows, and reflections can be applied to the image to allow for a natural integration of the underground facility model within the image before display.

[0063] The above user record section (122) can record preset inputs performed by a user wearing a user device in chronological order.

[0064] Here, the preset input may mean input through an external input device (touchpad, keyboard, mouse, controller, etc.) linked to the user device or an input device within the user device, such as voice recognition, gesture recognition, or head movement recognition.

[0065] The underground facility modeling update unit (123) can reflect the modified underground facility modeling displayed on the user device and output it to the user device when the user modifies the underground facility modeling displayed on the user device.

[0066] Here, modification of underground facility modeling using the user device may mean modification of some displays such as changes in the material texture of the model, crack indications, damage indications, and colors.

[0067] At this time, user information and modification time can be merged and output from the user device that performed the modification.

[0068] Through this, the name of the user who modified the underground facility model, the time of modification, etc. are recorded, so that other users can easily identify the modification details when outputting the same underground facility model.

[0069] The above-mentioned underground facility notification unit (124) can display on the user device at least one notification among the underground facility inspection cycle, parts replacement period, scheduled construction date, and construction completion date input by a previous user who input and modified underground facility modeling using a user device or a user who shares a server (not shown) in response to preset conditions.

[0070] Here, the above-described preset conditions refer to temporal conditions and spatial conditions, and the temporal conditions may be such that when the user has set up notifications for the underground facility modeling, notifications such as the construction date and parts replacement date for the underground facility modeling may be displayed 1 day in advance, 10 days in advance, etc.

[0071] In addition, the above spatial conditions mean a case where the underground facility modeling is displayed using the user device, and the latest inspection date of the underground facility, the scheduled future construction date, etc. can be displayed on the user device.

[0072] The above matching unit (125) can perform matching between real-time captured images captured through a user device and displayed underground facility modeling.

[0073] Here, the above matching part (125) is described in more detail with reference to FIG. 5 and FIG. 8.

[0074] Referring to the above FIGS. 5 and 8, the alignment unit (125) is a screen where actual underground facilities corresponding to the underground facility modeling are displayed in a real-time image captured through a user device, as shown in FIG. 5, and if the location on the screen is not correct, the alignment of the underground facility modeling in the image can be performed by moving the location through an input from the user device as shown in FIG. 8.

[0075] Meanwhile, the display unit (120) can individually store the collected images by applying a time-lapse effect, which is a process of changing images in chronological order, while performing underground facility work as shown in FIGS. 6 and 7 using images captured by a user device.

[0076] Through this, the progress of the work can be recorded and shared as a video even when not at the actual work site.

[0077] Referring back to FIG. 1, the collaboration unit (130) can perform collaboration by sharing and transmitting / receiving at least one of real-time video, voice, text, image, 3D object, model (modeling), and video data between users (user devices or preset user terminals) using wireless communication.

[0078] Here, the process of performing collaboration between users using a user device or user terminal is performed in the same manner as a conventional video call, but multiple users can participate, and during a video call, video data and voice data are shared between each user through a user device or user terminal, so that efficient and accurate work and supervision can be performed simultaneously when working on underground facilities.

[0079] The above sensor unit (140) senses user environment information by providing a temperature measurement sensor, a harmful gas concentration measurement sensor, and a humidity measurement sensor at preset locations of the user device, and can further measure user health information by attaching at least one medical sensor among a thermal imaging camera, an electrocardiogram sensor, an oxygen saturation sensor, and a body temperature sensor.

[0080] Here, the above preset location may be moved depending on the type of user device, but it is ideal for the measurement sensor (meaning a temperature measurement sensor, a harmful gas concentration measurement sensor, and a humidity measurement sensor) to be installed at the nose bridge (an area close to the nose) close to the user's respiratory organ based on smart glasses.

[0081] Additionally, the medical sensor may be located inside the smart glasses so as to come into contact with the user's face.

[0082] For example, the medical sensor may be installed on a part of the user's face, such as the nose or the back of the ear, when installed on smart glasses to perform sensing.

[0083] At this time, the sensor unit (140) is described in more detail with reference to FIG. 3.

[0084] Referring to the above drawing 3, the sensor unit (140) may include a sensor information collection unit (141), a sensor error measurement unit (142), and a data transmission unit (143).

[0085] The above sensor information collection unit (141) can collect the supply voltage, consumption current, and output voltage supplied to the above-mentioned measurement sensor.

[0086] More specifically, the sensor unit (140) can collect the supply voltage, consumption current, and output voltage supplied from the power source by further providing a voltage measuring sensor on the power supply line and output line of the measuring sensor.

[0087] At this time, the measurement sensor means one of the temperature measurement sensor, the harmful gas concentration measurement sensor, and the dust amount measurement sensor mentioned in the sensor unit (140), and an identical sensor (a sensor of the same model) within a preset radius from the measurement sensor may be provided as a comparison sensor.

[0088] Here, the preset radius is ideally located within a radius of 10 mm to 500 mm from the center position of the measurement sensor, and it can be judged that the closer it is to the measurement sensor, the higher the accuracy when compared.

[0089] The above sensor error measurement unit (142) can extract a comparison target time, which is a time when the rate of change in the measured value of the sensing data collected from the measurement sensor exceeds a preset rate.

[0090] Here, the preset ratio can be set to a ratio of 25% to 30%, and the setting can be changed from a minimum of 10% to a maximum of 90% depending on the type of measurement sensor and the surrounding environment.

[0091] In addition, the method for calculating the rate of change in the measured value of the sensing data is as follows: the measured value T measured at time t2 within the time range of the sensing data collected by the sensor information collection unit (141) t2 t in a The measurement value T measured at time t1, which is a time interval t1 T is the value minus t1 After dividing the measurement value, multiplying it by 100 to obtain the absolute value, the rate of change in the measurement value can be calculated.

[0092] For example, if the time interval set by the user is 2 seconds, the total measurement time from the start of operation of the current measurement sensor, the temperature sensor, to the end is 2000 seconds, and the measurement value measured at 1520 seconds is 21℃ and the measurement value measured at 1518 seconds is 31.6℃, the measurement value change rate can be calculated as approximately 36.2% (100*(21-28.6) / 21). At this time, if the ratio set by the user is 30%, 1518 seconds can be extracted as the comparison target time because the measurement value change rate exceeds 30%.

[0093] Meanwhile, if there is no time at which the rate of change in the measured value exceeds a preset rate and is therefore not extracted, the sensor error measurement unit (142) can extract sensing data corresponding to n arbitrary times within the sensing data as comparison target times and transmit the extracted data to the monitoring unit (160).

[0094] Here, the method of extracting random n visual points within the sensing data can be performed using a program using a programming language, and the programming language can mean Python, Java, C language, C++, JavaScript, Go, Ruby, Swift, Kotlin, PHP, C# (C Sharp), etc.

[0095] For example, a method for extracting n random times from sensing data using Python can be done by setting the time range in which the sensor operates in the sensing data using the Random module and the Datetime module and then extracting n random times, or by using the Numpy module.

[0096] Additionally, another method for randomly extracting n time points within the sensing data is to apply the Fisher-Yates shuffle algorithm to the time range of the sensing data to randomly extract n time points within the sensing data.

[0097] Meanwhile, among the n points of time extracted from the sensor error measurement unit (142), n means a natural number that may vary depending on the operating environment set by the user, and the meaning of the arbitrary point of time may mean a point of time extracted in a random format without regularity from the entire time collected from the sensing data.

[0098] The above data transmission unit (143) can periodically transmit the comparison target time of the sensor error measurement unit (142) to the monitoring unit (160).

[0099] In addition, the data transmission unit (143) can transmit data including the supply voltage, consumption current, and output voltage of the measurement sensor collected by the sensor information collection unit (141) to the monitoring unit (160).

[0100] Through the above process, the sensor unit (140) can transmit sensing data, which is a set of measurement values ​​collected from a plurality of measurement sensors, to a user terminal, and the user can distinguish a section in which the change rate of the sensing data does not exceed a preset change rate in response to a set time interval as a 'standard state section', and distinguish a section in which the change rate exceeds the preset change rate as an 'abnormal state section'.

[0101] Here, when the 'abnormal state section' of the sensing data collected from the sensor unit (140) using the input device including the user terminal exceeds the maximum time range of the 'abnormal state section' set by the user or the number of times the 'abnormal state section' has occurred exceeds a certain number of times, a notification such as work stoppage, location movement, or emergency departure from the current location can be set to be automatically displayed on the user device where the sensor is located. Through this, a quick response can be performed to the occurrence of an abnormal phenomenon.

[0102] For example, if the upper limit of the temperature measurement sensor provided in the user device is set to 30℃, the lower limit is set to 20℃, and if the rate of change exceeds 20% and it is set as an abnormal state section, if the temperature measurement value measured at 149 seconds is 24℃, the temperature measurement value measured at 150 seconds is 29℃, and the temperature measurement value measured at 151 seconds is 31℃, the rate of change for the 149 to 150 second section can be calculated as approximately 20.83% (100*(29-24) / 24). At this time, the 149 to 150 second section can be classified as an 'abnormal state section' because the rate of change per second exceeds 20%. In addition, the rate of change per second for the above 150 to 151 second section is calculated as approximately 6.89℃ (100*(31-29) / 29), which means that the rate of change per second does not exceed 20%. However, since the temperature measurement value measured at 151 seconds exceeded the upper limit of 30℃ set by the user, it can be classified as an 'abnormal condition section'.

[0103] At this time, if an abnormal condition section such as the above occurs, a work area confirmation notification may be automatically displayed on the safety manager's user device or user terminal.

[0104] Referring again to Figure 1, the risk assessment unit (150) can assess the risk of a user wearing the user device using the information collected from the sensor unit (140).

[0105] Here, the risk assessment unit (150) is described in more detail with reference to FIG. 4.

[0106] Referring to Fig. 4, the risk judgment unit (150) may include a fever judgment unit (151), an air quality judgment unit (152), a fire judgment unit (153), and a health abnormality judgment unit (154).

[0107] The above fever determination unit (151) can determine whether or not a fever occurs based on the user's body temperature measurement value collected by the sensor unit (140).

[0108] Here, the standard range for determining whether or not a fever has occurred can be set by the user or administrator, and the range can generally be set to 36℃ to 37℃, and the range can be freely changed depending on the work environment and the user's condition.

[0109] The above air quality judgment unit (152) can judge the air condition in response to the harmful gas concentration measurement value collected from the harmful gas concentration measurement sensor provided in the sensor unit (140).

[0110] Here, the above-mentioned standby state can be classified as normal if the concentration of harmful gases measured at the current location of the user device is within the allowable range, and as dangerous if it exceeds the allowable range.

[0111] More specifically, a case where the allowable concentration range of each hazardous gas measured by a hazardous gas concentration measuring sensor capable of measuring multiple hazardous gases is exceeded is classified as a risk, and the allowable concentration range can be set based on the range according to safety regulations.

[0112] The above fire judgment unit (153) can determine whether there is a fire using a thermal imaging camera provided in the user device.

[0113] Here, the thermal imaging camera is installed on the outer surface of the user device, but is installed in the same direction as the user's front line of sight, so that the presence of a fire can be determined based on the temperature, heat pattern, and color mapping caused by the fire.

[0114] For example, a fire in an image captured by the thermal imaging camera can be determined to be a fire because the flames generally appear in a pointed shape and a heat pattern is shown in which a zone of heat emission is formed around a high temperature area.

[0115] Additionally, thermal imaging cameras visualize heat by mapping it into colors, so higher temperatures are displayed in bright colors such as red and orange, and lower temperatures are displayed in dark colors such as blue and black. Therefore, if the area where a fire has occurred is displayed in a bright color and the temperature at that point is measured to be similar to the temperature measured during a fire, it can be determined to be a fire.

[0116] The above health abnormality determination unit (154) can determine that the user's health condition is abnormal when the electrocardiogram sensor and oxygen saturation sensor provided in the user device are out of the standard range.

[0117] Here, the above reference range can be set based on the oxygen saturation and electrocardiogram of a normal person.

[0118] In addition, the above health abnormality determination unit (154) can receive electrocardiogram, body temperature, and oxygen saturation data from a wearable device (such as a smart watch) capable of wireless communication worn by the user and determine whether there is a health abnormality based on the data.

[0119] For example, the criteria for determining whether or not a user is in good health may be set based on the user's heart rate collected through electrocardiogram data being within the range of 60 to 100 beats per minute, body temperature within the range of 36.5 to 37.5℃, and oxygen saturation being 95% or higher. The normal range setting value for determining normal may change depending on the user's health status and the surrounding environmental conditions.

[0120] Meanwhile, if at least one of the fever, air quality, fire, and health problems judged by the risk assessment unit (150) is judged to be a risk, a risk notification and a risk user device number can be displayed on the user device of the user device wearer and the administrator to prevent safety accidents.

[0121] Referring again to FIG. 1, the monitoring unit (160) can receive sensing data collected from the sensor unit (140) and transmit the environmental status to a preset user terminal.

[0122] Here, the monitoring unit (160) uses the sensing data collected from the measurement sensor and the comparison sensor and the comparison target time to calculate the sensor failure probability (S) according to the following [Mathematical Formula 1] TR ) can be produced.

[0123] [Mathematical Formula 1]

[0124]

[0125] (Here, S TR is the probability of sensor failure, T ms1 is the measurement value of the measurement sensor at the first time, T ms2 is the measurement value of the measurement sensor at the second time, T ms3 is the measurement value of the measurement sensor at the third time, T msn is the measurement value of the measurement sensor at the nth time, T cst1 is the measurement value of the comparison sensor at the first time, T cst2 is the measurement value of the comparison sensor at the second time, T cs3 is the measurement value of the comparison sensor at the third time, T csn is the measurement value of the comparison sensor at the nth time, T msav is the average measurement value of the measurement sensor, and SE means the efficiency deviation of the measurement sensor.)

[0126] At this time, the average efficiency (SE) of the above measurement sensor av ) can be calculated according to the following [Mathematical Formula 2].

[0127] [Equation 2]

[0128]

[0129] (Here, SE av is the average efficiency of the measuring sensor, SE i refers to the initial efficiency of the measuring sensor)

[0130] The average efficiency of the above measurement sensor can be calculated according to the following [Mathematical Formula 3].

[0131] [Equation 3]

[0132]

[0133] (Here, W IN1 is the power supplied to the measurement sensor at the first time, W OUT1 The power measured when the measurement value sensed by the measurement sensor at the first time is output as an electrical signal, W IN2 is the power supplied to the measurement sensor at the second time, W OUT2 The power measured when the measurement value sensed by the measurement sensor at the second time is output as an electrical signal, W INn is the power supplied to the measurement sensor at the nth time, W OUTn It means the power measured when the measurement value sensed by the measurement sensor at the nth time is output as an electrical signal.

[0134] At this time, the n time points mentioned in the above [Mathematical Formula 1] to [Mathematical Formula 3] may mean the number of multiple time points or randomly extracted time points at which the rate of change in the measured value of the sensing data extracted from the sensor error measurement unit (142) exceeds a preset rate.

[0135] For example, when the user device is provided with a first temperature sensor provided as a measurement sensor and a second temperature sensor provided as a comparison sensor of the same model as the first temperature sensor within a radius of 10 mm, the input / output voltage of the first temperature sensor and the current consumption inside the measurement sensor can be displayed. At this time, when three times are extracted from the measurement sensor, and the voltages supplied during the extracted first to third times are 4.9 V, 5 V, and 5 V, and the measurement sensor is measured to consume currents of 1 mA, 0.9 mA, and 1.1 mA at the corresponding times and output as 10 mV, 9 mV, and 11 mV, the input power (W) supplied to the measurement sensor at the first time IN1) is 0.0049W (4.9V*0.001A), and the input power at the second time (W IN2 ) is 0.0045W (5V*0.0009A), and the input power at the third time point (W IN3 ) can be calculated as 0.0055W (5V*0.0011A). In addition, the output power (W) of the first time output from the measurement sensor out1 ) is 0.00001W (0.01V*0.001A), the output power of the second time (W OUT2- ) is 0.000081W (0.009V*0.0009A), the output power of the third visual (W OUT3 ) can be calculated as 0.0000121W (0.011V*0.0011A). Using this, the average efficiency (SE) of the above measurement sensor av ) can be calculated as 0.2(100 / 3*(0.00001 / 0.0049+0.000081 / 0.0045+0.0000121 / 0.0055)). At this time, the initial efficiency (SE) provided by the manufacturer of the measuring sensor being measured i ) is 0.2%, the efficiency deviation (SE) of the measuring sensor can be calculated as 0. Here, the average efficiency (SE) of the measuring sensor av ) is an efficient sensor the lower the output value is, and the higher the output value is than the initial efficiency, which is the efficiency calculated using the measurement values ​​collected from the manufacturer or initially measured, the higher the probability of failure in the electrical domain of the sensor. Meanwhile, the initial efficiency (SE) of the measurement sensor i ) is not provided, the initial efficiency (SE) is the efficiency measured before the user device is installed. i ) can be set and input to the monitoring unit (160).

[0136] In addition, when the user's body temperature set in the user device is set to a temperature of 36.5°C, sensing data according to the user's work measured from the first temperature sensor provided in the user device can be collected from the moment the user turns on and puts on the device until the user releases the device. At this time, when the average measurement value collected from the first temperature sensor is 36.4°C and three random times are extracted from the sensing data of the first temperature sensor using a sensing data processing program, the measurement values ​​measured at each time can be extracted as 37°C (485 seconds), 37.5°C (1893 seconds), and 36.2°C (2021 seconds). Here, when the measurement values ​​measured at the same time and each time from the second temperature sensor that operates simultaneously with the first temperature sensor are 37.2°C (485 seconds), 38°C (1893 seconds), and 36.5°C (2021 seconds), the failure probability (S) of the first temperature sensor TR ) can be calculated. That is, the failure probability (S) of the first temperature sensor TR ) can be calculated as approximately 0.00009%(Min((|37-37.2|+|37.5-38|+|36.2-36.5|) / (3*36.4)*100+0), 100).

[0137] As another example, the monitoring unit (160) calculates the sensor failure probability (S) according to the following [Mathematical Formula 1-2], which reflects the weight according to the distance between the measurement sensor and the comparison sensor. TR ) can be produced.

[0138] [Equation 1-2]

[0139]

[0140] (Here, D v refers to the distance weighting of the measurement sensor and the comparison sensor)

[0141] At this time, the distance value weight (D-) of the above measurement sensor and comparison sensor v) is applied in response to the weight set by the user, and the user can freely change and apply the weight.

[0142] For example, if the measurement sensor and the comparison sensor are provided in the user device as temperature measurement sensors of the same model, the distance weight (D) of the measurement sensor and the comparison sensor according to the distance (D) of the measurement sensor and the comparison sensor v ) can be calculated according to [Table 1] below.

[0143] [Table 1]

[0144]

[0145] At this time, the distance weight (D v ) is the same condition as the above example that does not reflect the distance weight (D v ) is the failure probability of the first temperature sensor (S TR ) can be calculated as approximately 0.00009%(Min((|37-37.2|+|37.5-38|+|36.2-36.5|) / (3*36.4)*1*100+0), 100).

[0146] Through the above process, the monitoring unit (160) uses the sensing data collected from the sensor unit (140) to determine the failure probability (S) of the sensor. TR ) can be produced.

[0147] In addition, the failure probability (S) calculated through the above monitoring unit (160) TR ) exceeds the preset ratio, a sensor failure notification, replacement notification, etc. can be transmitted to the user device and the administrator's user device.

[0148] Here, the failure probability (S TR ) can be set to 40% by default, and the user can change the ratio between 0 and 99% depending on the type and sensitivity of the sensor.

[0149] Through the above process, the augmented reality-based guide visualization system for safety inspection of underground facilities collects a plurality of pieces of information from inside and outside, displays an underground facility model according to the user's location through the user's device, and collaborates by sharing and transmitting data between users through wireless communication. In addition, the displayed underground facility model is modified, and the underground facility status is notified. In addition, environmental information and the health information of the user wearing the device are collected through a plurality of sensors installed in the user's device, and when a certain range is exceeded, a notification is sent to the user's device and the administrator's device, thereby preventing safety accidents. In addition, in order to minimize work interruptions due to notifications of failure or malfunction of the multiple sensors installed in the user's device, the failure probability of each sensor can be calculated and sensor failure can be diagnosed in advance.

[0150] According to one embodiment of the present invention, by modeling an underground facility and providing modeling data based on the location of a user terminal as an extended reality-based guide image, a worker can check the location and information of the underground facility.

[0151] Additionally, by sharing real-time images currently being collected using user terminals with experts and transmitting and receiving data, work efficiency and accuracy can be improved.

[0152] Additionally, by having all workers wear user terminals that perform wireless communication, it is possible to identify the worker's location, communicate, connect with experts, and collect and share information such as the current user's health status and environmental information, simplifying worker health management and work communication.

[0153] Although the embodiments of the present invention have been described with limited examples and drawings, the embodiments of the present invention are not limited to the embodiments described above, and various modifications and variations are possible based on this description by those skilled in the art to which the present invention pertains. Therefore, the embodiments of the present invention should be understood solely by the scope of the claims set forth below, and all equivalent or equivalent modifications thereof are deemed to fall within the scope of the present invention.

[0154]

[0155] 100: An augmented reality-based guide visualization system for safety inspection of underground facilities.

[0156] 110: Information Collection Department

[0157] 120: Display section

[0158] 121: Modeling output section

[0159] 122: User Logbook

[0160] 123: Underground facility modeling update

[0161] 124: Underground facility notification section

[0162] 125: Matching part

[0163] 130: Collaboration Department

[0164] 140: Sensor section

[0165] 141: Sensor information collection unit

[0166] 142: Sensor error measurement unit

[0167] 142: Data transmission unit

[0168] 150: Risk Assessment Department

[0169] 151: Fever judgment unit

[0170] 152: Air quality judgment unit

[0171] 153: Fire Judgment Department

[0172] 154: Health Abnormality Determination Department

[0173] 160: Monitoring Department

Claims

1. In an extended reality-based guide visualization system for safety inspection of underground facilities, the system provides a virtual image aligned with a real image to the user by having the user wear a wearable device, An information collection unit that collects external data including the above user information, underground facility modeling, location information, date information, and time information; A display unit that superimposes the external data onto an image captured from the user device and outputs it; A collaboration unit that performs collaboration by sharing and transmitting at least one of real-time video, voice, text, image, 3D object, model, and video data between users using wireless communication; A sensor unit that senses user environment information by providing a temperature measurement sensor, a harmful gas concentration measurement sensor, and a humidity measurement sensor at preset locations of the user device, and measures user health information by attaching at least one medical sensor among a thermal imaging camera, an electrocardiogram sensor, an oxygen saturation sensor, and a body temperature sensor; A risk assessment unit that assesses the risk level of the user using the information collected from the sensor unit; and A monitoring unit that receives sensing data collected from the above sensor unit and transmits the environmental status to a preset user terminal; Including, The above sensor part, A sensor information collection unit that collects the supply voltage, consumption current, and output voltage supplied to the above measurement sensor; A sensor error measurement unit that extracts a comparison target time, which is a time when the rate of change in the measured value of the sensing data exceeds a preset rate; and A data transmission unit that periodically transmits the comparison target time of the above sensor error measurement unit to the above monitoring unit; The above sensor error measurement unit, An extended reality-based guide visualization system for safety inspection of underground facilities, characterized in that, if there is no time exceeding the above-mentioned preset ratio, sensing data corresponding to n random times in the sensing data is extracted as a time to be compared.

2. In paragraph 1, The above display unit, A modeling output unit that outputs the underground facility modeling corresponding to the user location information and the video being filmed; A user recorder that records preset inputs performed by a user wearing the user device in chronological order; An underground facility modeling update unit that reflects and outputs the modified underground facility modeling when the user modifies the underground facility modeling displayed on the user device; An underground facility notification unit that displays at least one notification among the underground facility inspection cycle, parts replacement period, and construction schedule entered by a previous user or a user sharing the server on the user device in response to preset conditions; and A matching unit that performs matching between the underground facility modeling output to the user device and the real-time captured image; An extended reality-based guide visualization system for safety inspection of underground facilities, characterized by including:

3. In paragraph 1, The above risk assessment unit is, A fever determination unit that determines whether or not a fever occurs based on the user's body temperature measurement value collected by the sensor unit; An air quality judgment unit that judges the atmospheric condition in response to the harmful gas concentration measurement value collected from the above sensor unit; A fire judgment unit that determines whether there is a fire using a thermal imaging camera provided in the user device; and A health abnormality determination unit that determines the user's health status when the electrocardiogram sensor and the oxygen saturation sensor are out of the reference range; An extended reality-based guide visualization system for safety inspection of underground facilities, characterized by including:

4. In paragraph 1, The above sensor part, A comparison sensor is provided with an identical sensor within a preset radius from at least one of the temperature measurement sensor, the harmful gas concentration measurement sensor, and the humidity measurement sensor. The above monitoring unit, Using the sensing data collected from the above measurement sensor and the comparison sensor and the comparison target time, the sensor failure probability (S) is calculated according to the following [Mathematical Formula 1] TR ) is produced, [Mathematical Formula 1] (Here, S TR is the probability of sensor failure, T ms1 is the measurement value of the measurement sensor at the first time, T ms2 is the measurement value of the measurement sensor at the second time, T ms3 is the measurement value of the measurement sensor at the third time, T msn is the measurement value of the measurement sensor at the nth time, T cst1 is the measurement value of the comparison sensor at the first time, T cst2 is the measurement value of the comparison sensor at the second time, T cs3 is the measurement value of the comparison sensor at the third time, T csn is the measurement value of the comparison sensor at the nth time, T msav is the average measurement value of the measurement sensor, and SE means the efficiency deviation of the measurement sensor.) The efficiency deviation (SE) of the above measurement sensor is calculated according to the following [Mathematical Formula 2], [Equation 2] (Here, SE av is the average efficiency of the measuring sensor, SE i refers to the initial efficiency of the measuring sensor) An extended reality-based guide visualization system for safety inspection of underground facilities, characterized in that the average efficiency of the above measurement sensor is calculated according to the following [Mathematical Formula 3]. [Equation 3] (Here, W IN1 is the power supplied to the measurement sensor at the first time, W OUT1 The power measured when the measurement value sensed by the measurement sensor at the first time is output as an electrical signal, W IN2 is the power supplied to the measurement sensor at the second time, W OUT2 The power measured when the measurement value sensed by the measurement sensor at the second time is output as an electrical signal, W INn is the power supplied to the measurement sensor at the nth time, W OUTn It means the power measured when the measurement value sensed by the measurement sensor at the nth time is output as an electrical signal.

5. In paragraph 4, The above monitoring unit, By further reflecting the weights according to the distance of the above measurement sensor and comparison sensor, the sensor failure probability (S) is calculated according to [Mathematical Formula 4] and [Table 1] below. TR ) is a guide visualization system based on extended reality for safety inspection of underground facilities. [Equation 4] (Here, D v refers to the distance weighting of the measurement sensor and the comparison sensor) [Table 1]

Images