Digital tutoring method for remote aircraft maintenance training of learners

Abstract

The present invention provides a digital tutoring method for remote aircraft maintenance education and training of a learner, comprising: a step in which an object recognition module recognizes or identifies the aircraft equipment or its parts, which are target objects, using a pre-trained object recognition learning model, and generates corresponding 3D modeled content based on an image of the target object; a step in which an authoring tool module registers the content and creates a training scenario using the registered content according to user input; a step in which a wearable device worn on the learner's head recognizes the aircraft equipment and its parts in real-time without markers using the object recognition learning model; and a step in which a field local server checks the progress status according to the training scenario generated by the authoring tool module based on the object recognized through the wearable device.

Description

Technology Field

[0001] The present invention relates to a digital tutoring method for remote aircraft maintenance education and training of a learner. Background Technology

[0002] Maintenance of aircraft, particularly military aircraft, requires skilled technical expertise, and since aircraft maintenance is directly linked to operational capabilities, maintenance education and training must be performed repeatedly and accurately.

[0003] Traditionally, aircraft maintenance training has been conducted offline using paper manuals led by instructors, or has utilized video-based CBT (Computer Based Training).

[0004] These existing educational methods have significant limitations in recognizing or evaluating a learner's current learning status or learning accuracy in real time.

[0005] To address these issues, technologies that provide virtual training using immersive content such as Virtual Reality (VR) or Augmented Reality (AR) have recently been introduced; however, there are limitations in that they cannot recognize when a learner is incorrectly performing the content or procedures according to the correct maintenance scenario, nor can they provide feedback accordingly.

[0006] Therefore, there is a growing need for technologies to resolve these problems. Prior art literature

[0007] (Patent Document 0001) KR 10-2023-0134677 A The problem to be solved

[0008] The present invention aims to provide a digital tutoring method for aircraft maintenance education and training that enables an educator to create various training scenarios for aircraft maintenance education and training, and to provide customized feedback to learners based on the condition of equipment or its parts occurring during education or maintenance training according to the training scenarios. means of solving the problem

[0009] To solve the above problem, the present invention provides a digital tutoring method for remote aircraft maintenance education and training of a learner, comprising: an object recognition module, using a pre-trained object recognition learning model, to recognize or identify the target object, which is the aircraft equipment or its parts, and to generate corresponding 3D modeled content based on an image of the target object; a writing tool module, registering the content and creating a training scenario using the registered content according to user input; a wearable device worn on the learner's head, using the object recognition learning model, to recognize the aircraft equipment and its parts in real-time without markers; and a field local server, checking the progress status according to the training scenario generated by the writing tool module based on the object recognized through the wearable device.

[0010] According to one embodiment, the control server may further include the step of monitoring the progress according to the training scenario through a plurality of the field local servers.

[0011] According to one embodiment, the sensor module of the field local server may further include the step of predicting a virtual sensor value representing the state of the aircraft equipment based on the combination relationship of the parts recognized by the object recognition learning module.

[0012] According to one embodiment, each of the plurality of field local servers checks the progress status according to the training scenario for different aircraft equipment, and the step of predicting the virtual sensor value is such that the sensor module of the first field local server among the plurality of field local servers can predict the virtual sensor value based on the combination relationship of the parts being processed by the first field local server and, in addition, the combination relationship of the parts being processed by another second field local server among the plurality of field local servers.

[0013] According to one embodiment, the combination relationship of the parts may include at least one of the number of the combined parts, the positional accuracy of the combined parts, the combination structure of the combined parts, the connection path between the parts, and whether the combination order required in the training scenario matches.

[0014] According to one embodiment, the step of predicting the virtual sensor value is such that the sensor module of the first field local server can predict the virtual sensor value by dynamically adjusting a first weight according to the progress of the second field local server.

[0015] According to one embodiment, the field local server may further include a step of calculating the accuracy of the step-by-step procedure execution of the training scenario so as to enable evaluation of the learner according to the training scenario.

[0016] According to one embodiment, the accuracy of the procedure execution can be calculated based on coupling accuracy as an indicator based on the degree of agreement between the coupling sequence, position, and direction of the parts at each stage of the training scenario, and coupling reliability as an indicator based on the rate of omission of coupling of the coupling means between the parts.

[0017] According to one embodiment, the field local server may further include the step of outputting a safety risk alarm through the wearable device based on the coupling accuracy or fastening reliability of the parts recognized by the object recognition learning module.

[0018] According to one embodiment, the sensor module of the first field local server can predict the virtual sensor value by applying a second weight, which represents the accuracy of the procedure execution for the training scenario of the second field local server, to the first weight.

[0019] According to one embodiment, the device may further include a large-scale language model that has been pre-learned a manual for the aircraft equipment and its parts, and the wearable device may further include the step of providing an answer corresponding to a question from a learner to the learner using the large-scale language model. Effects of the invention

[0020] The digital tutoring method for aircraft maintenance education and training for learners according to the present invention allows an educator to create various training scenarios for aircraft maintenance education and training, and to provide customized feedback to learners based on the condition of equipment or its parts that occurs during education or maintenance training according to the training scenarios. Brief explanation of the drawing

[0021] FIG. 1 is a configuration diagram of a digital tutor system for aircraft maintenance education and training according to one embodiment of the present invention. FIG. 2 is an example of an aircraft that is a subject of training using a digital tutor system according to one embodiment of the present invention. FIG. 3 is an example diagram of the operation of an object recognition module and a authoring tool module according to an embodiment of the present invention. FIG. 4 is a step-by-step flowchart of a digital tutoring method for aircraft maintenance education and training according to one embodiment of the present invention. FIG. 5 is a diagram showing the relationship between sensor modules of the first and second field local servers for different fields according to an embodiment of the present invention. FIG. 6 is a step-by-step flowchart of a digital tutoring method including the step of calculating a virtual sensor value according to one embodiment of the present invention. FIG. 7 is a step-by-step flowchart of a digital tutoring method including a step of calculating the accuracy of the step-by-step procedure execution of a training scenario according to one embodiment of the present invention. FIG. 8 is a configuration diagram of a system in which a learner wearing a wearable device uses a large-scale language model according to one embodiment of the present invention. FIG. 9 is a step-by-step flowchart of a digital tutoring method for aircraft maintenance education and training, including the step of answering a learner's question using a wearable device utilizing a large-scale language model according to one embodiment of the present invention. Specific details for implementing the invention

[0022] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate easy understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0023] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0024] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0025] In this specification, terms such as “comprising” or “having” are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not excluding in advance the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0027] FIG. 1 is a configuration diagram of a digital tutor system for aircraft maintenance education and training according to one embodiment of the present invention.

[0028] As illustrated in FIG. 1, a digital tutor system for remote aircraft maintenance education and training (hereinafter abbreviated as "digital tutor system") (1) according to one embodiment of the present invention comprises: an object recognition module (41) that generates 3D modeled content corresponding to an image of aircraft equipment or its parts based on an image of an aircraft equipment or its parts using a pre-trained object recognition learning model; an authoring tool module (42) that registers the content and creates a training scenario using the registered content according to user input; a wearable device (20) that is worn on the head of a learner and recognizes aircraft equipment and its parts in real-time without markers using the object recognition learning model; a field local server (10) that checks the progress status according to the training scenario generated by the authoring tool module (42) based on the object recognized through the wearable device (20); and a control server (30) that monitors the progress status according to the training scenario through a plurality of field local servers (10). It can be included.

[0029] This digital tutor system (1) allows an educator to create various training scenarios for aircraft maintenance education and training using an authoring tool module (42), and the field local server (10) can recognize aircraft equipment or its parts through a wearable device (20) worn on the learner's head and provide customized feedback to the learner based on the status of the equipment or its parts according to the training scenario.

[0030] Below, we will examine each component in detail.

[0032] The operating server (40) is a server for overall operation of the digital tutor system (1) according to one embodiment of the present invention, and may include an object recognition module (41) capable of recognizing or identifying an object using a previously trained object recognition learning model as illustrated in FIG. 1.

[0033] The object recognition module (41) may be a module that enables the learner, who is wearing a wearable device (20) on their head, to recognize or identify the target object they are looking at based on an object recognition learning model, which is an artificial intelligence model, from a video stream input through the wearable device (20) or a camera provided therein when the learner looks at actual aircraft equipment or parts thereof in the real world.

[0034] The field local server (10) is linked with the object recognition module (41) and the wearable device (20) to recognize or identify the target object that the learner is looking at from the video input through the wearable device (20).

[0035] At this time, although the present invention is not specifically limited, it is preferable that the object recognition module (41) recognizes objects using a deep learning model such as YOLO (You Only Look Once), Faster R-CNN, or EfficientDet without using a separate marker predetermined during object detection and recognition.

[0036] The object recognized or identified by the object recognition module (41) is an aircraft equipment or a part thereof, and as shown in FIG. 2, it may be various devices including the engine of an aircraft (e.g., Surion helicopter) or a part included therein, and in this case, the aircraft equipment may be grouped into multiple units by aircraft zone as shown in the table below.

[0037] Zone 1 Forward fuselage Various components such as the nose door, tail landing gear, rotor system hydraulic control, and cockpit instruments Zone 2 central fuselage Main wheels, landing gear, fuel tank, structural hoist inside the passenger cabin, etc. Zone 3 Transition Auxiliary Power Unit Controller, APU Generator Control Unit, Battery, etc. Zone 4 Tail boom Tail rotor drive shaft, tail bumper shock absorber, intermediate gearbox oil temperature sensor, iron detector, etc. Zone 5 Vertical stabilizer / tail rotor Tail gearbox, tail rotor, horizontal and vertical stabilizers, etc. Zone 6 Upper deck Engine, main gearbox, main rotor, hydraulic system, alternator, auxiliary power unit, etc.

[0038] An object recognition learning model can be trained based on a dataset consisting of actual images or videos of aircraft equipment or its parts, wherein the dataset may consist of images taken in various environments (e.g., various lighting, angles, conditions, etc.) and labels for the equipment or its parts (e.g., part name, location, function, etc.).

[0039] In addition, an object recognition module (41) according to one embodiment of the present invention can generate corresponding 3D modeled content based on an image of an aircraft equipment or its parts, which is a target object, using an object recognition learning model.

[0040] Specifically, as illustrated in FIG. 3, the object recognition module (41) can generate 3D modeled content (I2) using an object recognition learning model for a 2D image (I1) of an input aircraft equipment or its parts.

[0041] At this time, the object recognition module (41) can recognize or identify the relevant parts, etc. using a pre-trained object recognition learning model, and based on this, can generate three-dimensional modeled content (I2) for the entire equipment or the parts included therein.

[0042] In addition, a digital tutor system (1) according to one embodiment of the present invention may include an authoring tool module (42) for creating a training scenario according to user input using three-dimensional modeled content (I2) generated by an object recognition module (41).

[0043] Specifically, the authoring tool module (42) can provide a predetermined GUI environment to the user through a fixed terminal such as a PC or a mobile terminal such as a tablet PC, and the learner can create a maintenance procedure or training scenario for education and training based on the 3D modeled content (I2) that is created and registered by the object recognition module (41) through user input.

[0044] The user can arrange the 3D modeled content (I2) of the target object in the steps and order of the training scenario through the GUI environment provided by the authoring tool module (42), set the combination relationship of the target object required for each step, set the standard time required for each step, or set the evaluation conditions for each step.

[0045] The training scenario produced by the authoring tool module (42) can be configured in a linear or tree-like (or branched) manner according to the flow of each step, and the training scenario produced in this way can be serialized and stored in a format such as XML or JSON.

[0046] The field local server (10) can store and prepare a training scenario generated according to user input entered through the terminal by linking with the authoring tool module (42) and / or the terminal, etc., and can check the progress status of whether maintenance education training by a learner at the field is proceeding normally according to the training scenario generated by using the prepared training scenario, or can evaluate the learner's maintenance education training using this.

[0047] In addition to this, the field local server (10) can provide the training scenario produced above to the learner through the wearable device (20) to guide the learner's maintenance education training to be carried out according to the training scenario.

[0048] In addition, a digital tutor system (1) according to one embodiment of the present invention may include a wearable device (20) worn on the head of a learner.

[0049] The wearable device (20) is a device that is worn on the learner's body, particularly on the head, to receive various inputs from the user or output various information to the user to interact with the user, and may be an HMD (Head-Mounted Display), smart glasses, MR device, etc.

[0050] The wearable device (20) can be connected to communicate with the field local server (10) or the control server (30), and when a learner wearing the wearable device (20) looks at actual aircraft equipment or parts thereof in the real world, the wearable device (20) can photograph the target object in the real world in real time using a camera provided therein and link with the object recognition module (41) through the field local server (10), thereby enabling the field local server (10) or the control server (30) to collect the current status of the target object, which is the aircraft equipment or parts thereof.

[0051] At this time, the wearable device (20) according to one embodiment of the present invention may include a TPU (Tensor Processing Unit) for recognizing or identifying a target object in conjunction with an object recognition module (41).

[0052] In addition, the wearable device (20) may receive from the field local server (10) or the control server (30), or output various information calculated by the wearable device (20) by overlaying it onto the learner's field of vision in an AR manner, or provide sound through a speaker or the like.

[0053] As a specific example, the wearable device (20) can provide instructions input by an educator to a learner in various forms, including screen or voice, through a local server (10) or a control server (30).

[0054] In addition, as described above, the wearable device (20) and / or the field local server (10) can recognize or identify aircraft equipment and its parts, which are target objects in the real world, from a video stream input in real time through the wearable device (20), and the field local server (10) can check the progress status of the maintenance education training currently being conducted by the learner according to the training scenario by using the target object recognized or identified by the wearable device (20) and the training scenario generated by the authoring tool module (42).

[0055] In addition, the local server (10) can provide various information related to the progress status to the wearable device (20) and output it in various forms including screens or voice to the learner.

[0056] Specifically, maintenance training for learners can be conducted at a site forming a designated space, and when maintenance training for aircraft equipment is conducted by a learner wearing a wearable device (20) at each site, the site local server (10) can check the progress status of whether the maintenance training conducted by the learner is proceeding according to the step-by-step sequence of the training scenario generated by the authoring tool module (42).

[0057] That is, the training scenario generated by the authoring tool module (42) can store the state in which at least one part is combined step by step in order, and the field local server (10) can compare this with the state of the target object currently recognized through the wearable device (20) so that the learner can verify whether the maintenance training currently in progress is proceeding accurately step by step according to the training scenario.

[0058] By checking whether the combination of parts by the learner is being carried out according to the sequence of a predetermined training scenario, and whether the parts are being fastened without omission using fastening means such as bolts at predetermined locations when combining parts, the field local server (10) can calculate the accuracy of the procedure execution as an indicator based on the degree of agreement of the order, location, and direction of combination of parts at each stage, and as an indicator based on the fastening reliability as an indicator based on the rate of omission of fastening means.

[0059] In addition, by using this, the field local server (10) can calculate a comprehensive evaluation score for the maintenance education training of the learner by determining whether the accuracy of the calculated procedure execution satisfies a standard accuracy level or higher within the execution time set for each stage of the training scenario.

[0060] Meanwhile, a digital tutor system (1) according to one embodiment of the present invention is connected to communicate with a plurality of field local servers (10) and can monitor whether maintenance training being conducted by learners at each field is proceeding according to a pre-made training scenario, and can also monitor evaluation scores for the maintenance training of learners at each field.

[0061] That is, the field local server (10) can correspond one-to-one with the site where the learner's maintenance training is taking place, and the field local server (10) and the control server (30) correspond in a many-to-one manner, so that the control server (30) can monitor the progress of the maintenance training being conducted by multiple learners at multiple sites through multiple field local servers (10), or the evaluation scores of the learners' maintenance training at each site.

[0062] In this case, if there are multiple learning or training sites for the learner, education or training may be conducted at each site by different aircraft zones, or by different equipment within the same aircraft zone.

[0063] For example, referring to the table above, education or training of learners regarding Zone 1 may be conducted at the first site, and education or training of learners regarding a zone other than Zone 1 may be conducted at the second site. Alternatively, education or training of learners regarding the engine of Zone 6 may be conducted at the first site, and education or training of learners regarding another device or part of the same Zone 6, such as the main gearbox, may be conducted at the second site.

[0065] Meanwhile, FIG. 4 is a step-by-step flowchart of a digital tutoring method for aircraft maintenance education and training according to one embodiment of the present invention.

[0066] As illustrated in FIG. 4, a digital tutoring method for aircraft maintenance education and training according to an embodiment of the present invention (hereinafter abbreviated as "digital tutoring method") comprises: a step (S10) in which an object recognition module (41) recognizes or identifies the aircraft equipment or its parts, which are the target objects, using a pre-trained object recognition learning model, and generates corresponding 3D modeled content based on an image of the target objects; a step (S20) in which an authoring tool module (42) registers the content and creates a training scenario using the registered content according to user input; a step (S30) in which a wearable device (20) worn on the head of a learner recognizes the aircraft equipment and its parts in real-time without markers using an object recognition learning model, and a field local server (10) checks the progress status according to the training scenario generated by the authoring tool module (42) based on the object recognized through the wearable device (20). Including step (S40), the educator can create various training scenarios for aircraft maintenance education and training, and can also provide customized feedback to the learner based on the condition of equipment or its parts that occurs during education or maintenance training according to the training scenario.

[0067] At this time, the control server (30) may further include a step (S50) of monitoring the progress according to a training scenario by multiple trainees through multiple field local servers (10).

[0069] Meanwhile, a field local server (10) according to one embodiment of the present invention may include a sensor module (111, 121) that predicts a virtual sensor value representing the state of the aircraft equipment based on the combination relationship of parts recognized by an object recognition learning model, as illustrated in FIG. 5.

[0070] Accordingly, as illustrated in FIG. 6, the digital tutoring method may further include a step (S35) in which the sensor module (111, 121) of the field local server 10) predicts a virtual sensor value representing the state of the aircraft equipment based on the combination relationship of the parts recognized by the object recognition learning module.

[0071] The sensor module (111, 121) of the field local server (10) can estimate various sensor values ​​by virtually estimating various information or status of at least some of the equipment or parts to be maintained without actual physical sensors, based on the connection relationship or fastening state of the aircraft equipment or parts thereof that are being trained by a learner at the field.

[0072] Specifically, from a video stream input in real time through the wearable device (20), the wearable device (20) and / or the field local server (10) can recognize or identify aircraft equipment and its parts, which are target objects in the real world, and the field local server (10) or its sensor module (111, 121) can estimate various sensor values ​​based on the connection relationship or fastening state of the aircraft equipment or its parts, which are the recognized or identified target objects.

[0073] Here, the virtual sensor values ​​may be electrical resistance indicating the presence of leakage current or resistance values ​​when connecting electrical components, pressure indicating leakage when joining hydraulic / pneumatic system components, leak detection indicating fluid leakage through gaps in the joints, i.e., seals, when joining components, thermal conductivity indicating thermal conductivity in coolers or heat exchange systems, or shaft rotation speed (RPM) of rotary drive means such as motors, but may be various measurement values ​​that can be detected during the assembly or completion of assembly of aircraft equipment or its components.

[0074] Specifically, the sensor module (111, 121) can estimate a virtual sensor value based on the number of combined parts, the positional accuracy of the combined parts, the combined structure of the combined parts, the connection path between the parts and / or whether it matches the required combination order in the training scenario.

[0075] The field local server (10) can determine whether the virtual sensor value estimated through the sensor module (111, 121) is within the normal range and whether the learner is proceeding normally according to the pre-generated training scenario.

[0076] Additionally, the field local server (10) can support education and training by transmitting the virtual sensor value to the wearable device (20) worn by the learner at the site to output it, thereby allowing the learner to check the condition of the equipment or parts to be maintained, and the field local server (10) can also support education and training by transmitting the virtual sensor value calculated by the sensor module (111, 121) to the control server (30), so that the educator can monitor the status of the maintenance education and training being conducted by the learner at the site through the control server (30).

[0077] Meanwhile, according to one embodiment of the present invention, a plurality of field local servers (10) corresponding to each of a plurality of sites can check the progress status according to the training scenario for different aircraft equipment currently undergoing maintenance training at each of the plurality of sites.

[0078] Specifically, as illustrated in FIG. 5, at the first site, a learner can perform maintenance training by maintaining parts (p1) of the first aircraft equipment (e1) that is subject to maintenance, and at the second site, a learner can perform maintenance training by maintaining parts (p2) of the second aircraft equipment (e2) that is subject to maintenance. At this time, the first site local server (11) can check the progress of the learner according to the first training scenario created at the first site, and the second site local server (12) can check the progress of the learner according to the second training scenario created at the second site.

[0079] At this time, the first site and the second site, and the learner of the first site and the learner of the second site are each independent, and the first sensor module (111) of the first site local server (11) can calculate various virtual sensor values ​​for the maintenance target by the learner of the first site, and likewise, the second sensor module (121) of the second site local server (12) can calculate various virtual sensor values ​​for the maintenance target by the learner of the second site.

[0080] However, even if the aircraft equipment subject to maintenance at the first site and the aircraft equipment subject to maintenance at the second site are equipment in different areas or different equipment in the same area, since the aircraft equipment subject to maintenance at the first site and the aircraft equipment subject to maintenance at the second site are related to the same aircraft, according to a preferred embodiment of the present invention, when the sensor module (111, 121) of one site local server (10) calculates a virtual sensor value, it is desirable to use not only the combination relationship of the parts of the aircraft equipment subject to maintenance at the first site, but also the combination relationship of the parts of the aircraft equipment subject to maintenance at the second site, which is another site.

[0081] That is, when maintenance training is being conducted simultaneously on equipment of different aircraft zones or equipment of the same aircraft zone at each of the different sites, cooperative maintenance training can be conducted among learners belonging to different sites. In this case, when the first sensor module (111) of the first site local server (11) calculates a virtual sensor value, it is desirable to base it on the combination relationship of the parts of the aircraft equipment subject to maintenance at the second site, in addition to the combination relationship of the parts of the aircraft equipment subject to maintenance at the first site. This is because, as mentioned above, the equipment in each zone within a single aircraft can influence each other even if they are different.

[0082] As a specific example, when the first sensor module (111) of the first field local server (11) calculates a virtual sensor value based on the combination relationship of the parts at the first field according to the number of parts currently combined at the first field, the positional accuracy of the combined parts, the combination structure of the combined parts, the connection path between the parts, and whether it matches the combination order required in the training scenario, it is desirable to also include the combination relationship of the parts at the second field according to the number of parts currently combined at the second field, the positional accuracy of the combined parts, the combination structure of the combined parts, the connection path between the parts, and whether it matches the combination order required in the training scenario.

[0083] However, by the sensor module (111, 121) of the field local server (10) according to one embodiment of the present invention calculating a virtual sensor value, the learner can receive the virtual sensor value through the wearable device (20), and thereby the learner can check the condition of the equipment to be maintained or its parts.

[0084] Additionally, the field local server (10) can use virtual sensor values ​​to verify whether maintenance training by the learner is proceeding normally according to a pre-generated training scenario, or the field local server (10) can use this to calculate an evaluation score for the learner's maintenance training. That is, the field local server (10) can pre-establish and store a normal range for at least one virtual sensor value for each step of the training scenario, and by comparing this with the current virtual sensor value calculated by the sensor module (111, 121) to determine whether the current virtual sensor value is within the normal range, it can verify whether maintenance training by the learner is proceeding normally according to a pre-generated training scenario, or calculate an evaluation score for the learner's maintenance training (for example, an evaluation score based on the number of virtual sensor values ​​outside the normal range and / or the degree to which they are outside the normal range).

[0085] At this time, if there is a significant difference in the progress of maintenance training according to the training scenario generated in each of the first and second sites, exceeding a pre-set standard range, when the first sensor module (111) of the first site local server (11) for the first site calculates a virtual sensor value, a distorted sensor value may be calculated due to the combination relationship of equipment or parts of the other second site that has a relatively low progress, and there is a problem that the progress of maintenance training at the other site that has a relatively high progress may be delayed by the maintenance training with a low progress.

[0086] Therefore, to resolve this problem, when the first sensor module (111) of the first field local server (11) according to one embodiment of the present invention calculates a virtual sensor value, it is desirable to dynamically adjust the first weight according to the progress of the maintenance training of the second field local server (12) according to the progress.

[0087] As a specific example, when the first sensor module (111) of the first field local server (11) calculates a virtual sensor value according to the number of combined parts at the first field, the positional accuracy of the combined parts, the combined structure of the combined parts, the connection path between the parts, and whether it matches the combination order required in the training scenario, the first weight (w1), for example, a value between 0 and 1 corresponding to the progress according to the training scenario of the second field or the second learner, can be calculated by multiplying the virtual sensor value (S1) calculated by the first sensor module (111) itself (see Equation 1 below).

[0088] [Mathematical Formula 1]

[0089] Virtual sensor value (S1′) for a device or its component at the first site = Virtual sensor value (S1) calculated by the first sensor module (111) itself × First weight (w1) corresponding to the progress according to the training scenario at the second site

[0090] Here, the first weight (w1) is a progress according to the training scenario of the second field calculated by the second field local server (12), and according to a preferred embodiment, the first weight (w1) corresponding to the progress may have a discrete value, and the progress and the first weight (w1) may be stored in the first field local server (11) or the second field local server (12) in the form of a lookup table.

[0091] In addition, according to a preferred embodiment, when the first sensor module (111) of the first field local server (11) calculates a virtual sensor value (S1′), it is preferable to also apply a second weight (w2) representing the accuracy of the step-by-step procedure execution up to the present according to the training scenario of the second field calculated by the second field local server (12), as shown in Equation 2 below (see Equation 2 below).

[0092] For example, the second weight (w2) may have a value between 0 and 1, which is the arithmetic mean of the accuracy of step-by-step procedure execution according to the training scenario of the second field, or the average of the accuracy with different weights applied according to the step.

[0093] [Mathematical Formula 2]

[0094] Virtual sensor value (S1′) for a device or part thereof at the first site = Virtual sensor value (S1) calculated by the first sensor module (111) itself × First weight (w1) corresponding to the progress according to the training scenario at the second site × Second weight (w2) representing the accuracy of the step-by-step procedure execution up to the present according to the training scenario at the second site

[0095] Of course, here the first weight (w1) can have a larger value as the progress according to the training scenario of the second site increases, and the second weight (w2) can have a larger value as the accuracy of the procedure execution according to the training scenario of the second site increases.

[0097] Meanwhile, FIG. 7 is a step-by-step flowchart of a digital tutoring method including a step of calculating the accuracy of the step-by-step procedure execution of a training scenario according to one embodiment of the present invention.

[0098] As illustrated in FIG. 7, a digital tutoring method according to one embodiment of the present invention may further include a step (S45) of calculating the accuracy of the step-by-step procedure execution of the training scenario so that the field local server (10) can evaluate the learner according to the training scenario.

[0099] As described above, a field local server (10) according to one embodiment of the present invention can calculate the accuracy of the step-by-step procedure execution of a training scenario based on an indicator of the degree of agreement between the order, position, and direction of assembly between aircraft equipment or its parts being trained by a learner in the field, and an indicator of the degree of assembly reliability based on the rate of omission of assembly of the assembly means.

[0100] At this time, the present invention is not limited thereto, and the field local server (10) can use the coupling accuracy or the fastening reliability to output a safety risk alarm through a wearable device (20) worn by a learner at the site when the coupling accuracy is less than a preset first threshold value or when the fastening reliability is less than a preset second threshold value.

[0101] As illustrated in FIG. 7, a digital tutoring method according to one embodiment of the present invention may further include a step (S46) in which a field local server (10) outputs an alarm of safety risk through a wearable device (20) based on the combination accuracy or fastening reliability of parts recognized by an object recognition learning module.

[0102] Since the parts of the aircraft equipment may be somewhat heavy and the assembly of these parts may not meet the standards, learners undergoing maintenance training may suffer injuries, etc., it is desirable for the field local server (10) to output a visual or auditory alarm through a wearable device (20) so that learners can recognize such problems in advance.

[0104] Meanwhile, FIG. 8 is a configuration diagram of a system in which a learner wearing a wearable device uses a large-scale language model according to one embodiment of the present invention.

[0105] As illustrated in FIG. 8, a digital tutor system (1) according to one embodiment of the present invention may include a Large Language Model (LLM) (112) that has been pre-learned various manuals related to aircraft equipment and its parts (e.g., maintenance manuals, technical documents, parts descriptions, etc.), and may provide a corresponding answer to a learner through the wearable device (20) in accordance with a question input by a learner wearing the wearable device (20).

[0106] At this time, the field local server (10) or the operating server (40) may include a large-scale language model (112), and the wearable device (20) may interact with the large-scale language model (112) to enable voice-based question-and-answer with the learner.

[0107] Specifically, the learner can input questions into the wearable device (20) in voice during maintenance training for aircraft equipment or its parts, and the wearable device (20) is linked with the large-scale language model (112) of the field local server (10) to provide answers to the questions to the learner, and it is desirable to provide a natural interface through STT (Speech-To-Text) or TTS (Text-To-Speech).

[0108] FIG. 9 is a step-by-step flowchart of a digital tutoring method for aircraft maintenance education and training, including the step of answering a learner's question using a wearable device utilizing a large-scale language model according to an embodiment of the present invention. As illustrated in FIG. 9, the digital tutoring method according to an embodiment of the present invention may further include the step (S36) in which the wearable device (20) uses a large-scale language model to provide an answer to the learner corresponding to a question from the learner.

[0110] Preferred embodiments of the present invention have been described in detail above with reference to the drawings. The description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without changing the technical concept or essential features of the present invention.

[0111] Accordingly, the scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning, scope, and equivalent concepts of the claims should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0112] 1: Digital Tutor System 10: On-site local server 111, 121: Sensor module 112: LLM 20: Wearable devices 21, 22: HMD 30: Control Server 40: Production Server 41: Object Recognition Module 42: Authoring Tool Module

Claims

Claim 1 A digital tutoring method for remote aircraft maintenance education and training of a learner, comprising: an object recognition module, using a pre-trained object recognition learning model, to recognize or identify the target object, which is the aircraft equipment or its parts, and to generate corresponding 3D modeled content based on an image of the target object; an authoring tool module, registering the content and creating a training scenario using the registered content according to user input; a wearable device worn on the learner's head, using the object recognition learning model, to recognize the aircraft equipment and its parts in real-time without markers; and a field local server, checking the progress status according to the training scenario generated by the authoring tool module based on the object recognized through the wearable device. Claim 2 A digital tutoring method for remote aircraft maintenance education and training of a learner, characterized in that, in claim 1, the control server further includes the step of monitoring the progress according to the training scenario through a plurality of the field local servers. Claim 3 A digital tutoring method for remote aircraft maintenance education and training of a learner, characterized in that, in claim 1, the sensor module of the field local server further includes the step of predicting a virtual sensor value representing the state of the aircraft equipment based on the combination relationship of the parts recognized by the object recognition learning module. Claim 4 In claim 3, each of the plurality of field local servers checks the progress status according to the training scenario for different aircraft equipment, and the step of predicting the virtual sensor value is characterized in that the sensor module of the first field local server among the plurality of field local servers predicts the virtual sensor value based on the combination relationship of the parts being processed by the first field local server and, in addition, the combination relationship of the parts being processed by another second field local server among the plurality of field local servers. Claim 5 A digital tutoring method for remote aircraft maintenance education and training of a learner, characterized in that, in claim 4, the coupling relationship of the parts comprises at least one of the number of coupled parts, the positional accuracy of the coupled parts, the coupling structure of the coupled parts, the connection path between the parts, and whether the coupling sequence required in the training scenario matches. Claim 6 In claim 4, the step of predicting the virtual sensor value is characterized in that the sensor module of the first field local server predicts the virtual sensor value by dynamically adjusting a first weight according to the progress of the second field local server. This is a digital tutoring method for remote aircraft maintenance education and training of a learner. Claim 7 A digital tutoring method for remote aircraft maintenance education and training of a learner, characterized in that, in claim 6, the field local server further includes a step of calculating the accuracy of the step-by-step procedure execution of the training scenario so as to enable evaluation of the learner according to the training scenario. Claim 8 A digital tutoring method for remote aircraft maintenance education and training of a learner, characterized in that, in claim 7, the accuracy of the procedure execution is calculated based on coupling accuracy as an indicator based on the degree of agreement between the coupling sequence, position, and direction of the parts at each stage of the training scenario, and coupling reliability as an indicator based on the rate of omission of coupling of the coupling means between the parts. Claim 9 A digital tutoring method for remote aircraft maintenance education and training of a learner, characterized in that, in claim 8, the field local server further comprises the step of outputting a safety risk alarm through the wearable device based on the coupling accuracy or fastening reliability of the parts recognized by the object recognition learning module. Claim 10 A digital tutoring method for remote aircraft maintenance education and training of a learner, characterized in that, in claim 7, the sensor module of the first field local server predicts the virtual sensor value by applying a second weight representing the accuracy of the procedure execution for the training scenario of the second field local server to the first weight. Claim 11 A digital tutoring method for remote aircraft maintenance education and training of a learner, characterized in that, in claim 1, it further includes a large-scale language model that has pre-learned a manual for the aircraft equipment and its parts, and the wearable device uses the large-scale language model to provide an answer corresponding to a question from the learner to the learner.

Images