Xr task training system using interaction between real and virtual objects and the method of thereof

Abstract

An XR training system and method utilizing the interaction between a real object and a virtual object are disclosed. An XR training system utilizing the interaction between a real object and a virtual object according to an embodiment of the present invention comprises: a work tool pairing unit that pairs a preset wearable device with at least one work tool; a content input unit that receives at least one training content through a preset user terminal when the pairing between the wearable device and the work tool is completed through the work tool pairing unit; a virtual object display unit that displays a virtual part image corresponding to the training content inputted in the content input unit on a display module provided in the wearable device; a virtual training control unit that recognizes the motion of the work tool through a camera module provided in the wearable device and controls the virtual part image in response to the motion of the work tool; and a feedback data generation unit that generates training feedback data corresponding to the training content and transmits the training feedback data to the user terminal when the training content is terminated.

Description

Technology Field

[0001] The present invention relates to an XR-based job training system and method, and more specifically, to an XR training system and method utilizing the interaction between a real object and a virtual object that enables job training without the use of consumables by utilizing the interaction between a real object and a virtual object. Background Technology

[0002] VR (Vitrual Reality) technology provides real-world objects or backgrounds solely as CG (Computer Graphic) images, AR (Augmented Reality) technology provides virtual CG images on top of real object images, and MR (Mixed Reality) technology provides virtual objects mixed and combined with the real world. The aforementioned VR, AR, and MR are all sometimes simply referred to as XR (Extended Reality) technology.

[0003] In particular, devices equipped with AR technology recognize real-world spaces and objects in three dimensions through a camera and display real-world spaces and virtual objects on a screen. However, according to conventional technology, virtual objects could only be moved based on user touches recognized on the screen.

[0004] Virtual reality is fundamentally based on providing visual and auditory stimuli to the user, but it has limitations in stimulating other senses. In the context of training and entertainment, there may not be many instances where olfactory and gustatory stimuli are required, but instances where tactile stimuli are required occur relatively frequently. For example, when an object or structure is visually simulated in virtual reality, the user may attempt to grasp the object with their hands or lean against the structure; however, since the object or structure does not exist within the real environment, this can lead to confusion or accidents such as falling.

[0005] In particular, when conducting training through virtual reality, interacting with surrounding structures can be a major part of the training. For example, when conducting counter-terrorism training using virtual reality, leaning behind structures such as walls to use them as a means of concealment or defense can be a significant part of the training for the trainee. In such cases, if virtual reality is provided only as audiovisual stimuli, the trainee cannot interact with the structures, which inevitably leads to a decline in the quality of the training.

[0006] To solve these problems, prior art Korean Published Patent No. 10-2021-0108840 discloses an augmented reality experience system capable of interacting with 3D objects and a method for implementing the same, and Korean Registered Patent No. 10-2333768 discloses a deep learning-based hand recognition augmented reality interaction device and method in which a user interacts with a virtual object using the user's body in an augmented reality environment.

[0007] Therefore, there is a need for research on XR training systems and methods utilizing the interaction between real and virtual objects to induce users to manipulate virtual objects more naturally and to enhance the immersion of task training. The problem to be solved

[0008] The present invention aims to provide an XR training system and method utilizing the interaction between real objects and virtual objects, which can train work processes more effectively without consuming virtual objects provided as consumables, by recognizing the motion of a work tool paired with a wearable device and training a work process using a virtual object displayed on the wearable device.

[0009] In addition, the purpose is to provide an XR training system and method utilizing the interaction between real and virtual objects that can provide customized work guide information necessary for the work process to the trainee by analyzing the motion of the work tool recognized through the work process and generating feedback information necessary for the work process to provide to the trainee.

[0010] The problems that the present invention aims to solve are not limited to those mentioned above, and other problems that the present invention aims to solve that are not mentioned herein will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0011] An XR training system utilizing interaction between a real object and a virtual object according to an embodiment of the present invention comprises: a work tool pairing unit that pairs a preset wearable device with at least one work tool; a content input unit that receives at least one training content through a preset user terminal when the pairing of the wearable device and the work tool is completed through the work tool pairing unit; a virtual object display unit that displays a virtual part image corresponding to the training content inputted in the content input unit on a display module provided in the wearable device; a virtual training control unit that recognizes the motion of the work tool through a camera module provided in the wearable device and controls the virtual part image based on the motion of the work tool; and a training feedback generation unit that generates training feedback data corresponding to the training content and transmits the training feedback data to a user terminal when the training content is terminated.

[0012] Additionally, the content input unit is characterized by including a content calling unit that calls a list of available training content from a pre-specified learning content DB corresponding to a work tool paired with a wearable device, a content list providing unit that transmits the list of available training content called by the content calling unit to a user terminal, and a content setting unit that receives at least one of the training content included in the list of available training content from the user terminal.

[0013] In addition, the virtual training control unit is characterized by including a motion recognition unit that recognizes the motion of a work tool captured through a camera module provided in a wearable device, and an image control unit that modifies a virtual part image in real time in response to the motion of the work tool recognized by the motion recognition unit.

[0014] In addition, the virtual object display unit is characterized by displaying a work guide image included in the training content by overlaying it on a virtual part image.

[0015] In addition, the training feedback generation unit generates training feedback data based on a virtual part image at the time when the training content ends and motion of a work tool captured through a camera module provided in a wearable device, wherein the training feedback data includes work time, work accuracy, and motion feedback information of the work tool. Effects of the invention

[0016] According to the present invention, by recognizing the motion of a work tool paired with a wearable device and training a work process using a virtual object displayed on the wearable device, the work process can be trained more effectively without consuming the virtual object provided as a consumable.

[0017] In addition, by analyzing the motion of the work tool recognized through the work process and generating feedback information necessary for the process to provide to the work trainee, it has the effect of providing customized work guide information required for the work process to the trainee. Brief explanation of the drawing

[0018] FIG. 1 is a configuration diagram of an XR training system utilizing the interaction between a real object and a virtual object according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a content input section of an XR training system utilizing the interaction between a real object and a virtual object according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a virtual training control unit of an XR training system utilizing the interaction between a real object and a virtual object according to an embodiment of the present invention. FIGS. 4 and 5 are drawings for explaining a fault diagnosis unit of an XR training system utilizing the interaction between a real object and a virtual object according to an embodiment of the present invention. Specific details for implementing the invention

[0019] Specific details regarding the problem to be solved, the means for solving the problem, and the effects of the invention as described above are included in the embodiments and drawings to be described below. The advantages and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the accompanying drawings.

[0020] The scope of the present invention is not limited to the embodiments described below, and various modifications can be made by those skilled in the art within the scope of the technical essence of the present invention.

[0022] Hereinafter, the title of the invention, the present invention, will be explained in detail with reference to the attached FIG. 1.

[0023] FIG. 1 is a configuration diagram of an XR training system utilizing the interaction between a real object and a virtual object according to an embodiment of the present invention, FIG. 2 is a diagram explaining the content input unit of an XR training system utilizing the interaction between a real object and a virtual object according to an embodiment of the present invention, FIG. 3 is a diagram explaining the virtual training control unit of an XR training system utilizing the interaction between a real object and a virtual object according to an embodiment of the present invention, and FIG. 4 and FIG. 5 are diagrams explaining the fault diagnosis unit of an XR training system utilizing the interaction between a real object and a virtual object according to an embodiment of the present invention.

[0025] <Example 1>

[0026] Referring to FIG. 1, an XR training system (100) utilizing interaction between a real object and a virtual object according to an embodiment of the present invention may include a work tool pairing unit (110), a content input unit (120), a virtual object display unit (130), a virtual training control unit (140), and a training feedback generation unit (150).

[0028] More specifically, the work tool pairing unit (110) can pair at least one work tool with a designated wearable device.

[0029] For example, the wearable device may be provided as a head-mounted display module worn on the head of a work trainee, and the work tool may be provided in the form of a tool used in a work environment, and the wearable device and the work tool may include a communication module such as a Bluetooth module or a Zigbee module, and may be paired through the communication module.

[0030] The content input unit (120) can receive at least one training content through a designated user terminal when the pairing of the wearable device and the work tool is completed through the work tool pairing unit (110).

[0031] For example, the user terminal may be provided with a smartphone, tablet PC, etc., carried by a job trainee, and when the user terminal is connected to a pre-designated program, the pairing status of the wearable device and the job tool may be checked through the user terminal.

[0032] In addition, when the pairing state of the wearable device and the work tool is completed through the user terminal, a list of work content that can be performed using the work tool may be displayed on the user terminal.

[0033] Accordingly, the content input unit (120) can receive training content that can be worked on using the work tool through the user terminal.

[0034] At this time, the content input unit (120) may receive information about the external training environment in which the training content will be virtually performed through the user terminal.

[0036] Meanwhile, the virtual object display unit (130) can display a virtual part image corresponding to the training content input to the content input unit (120) (122) on a display module provided in the wearable device.

[0037] For example, if the training content entered into the content input unit (120) is content that releases a valve provided in an arbitrary work environment, the work tool is a tool used to release the valve, and the virtual part image can be displayed as a valve provided in the virtual work environment.

[0038] Accordingly, the above-mentioned job trainer can perform virtual job training using the above-mentioned job tool on a virtual part image displayed on the wearable device.

[0040] The virtual training control unit (140) can recognize the motion of the work tool through a camera module provided in the wearable device and control the image of the work part based on the motion of the work tool.

[0041] For example, the motion of the work tool recognized through the camera module can be recognized as a motion of attaching the work tool to the valve, which is the virtual part image, to release the valve, and a motion of rotating the work tool to release the valve when the work tool is attached to the valve.

[0042] At this time, the virtual training control unit (140) can recognize the motion of the work tool by comparing the motion learning image stored in the pre-designated learning content DB in correspondence with the training content with the motion of the work tool, and control the virtual part image in correspondence with the motion of the work tool.

[0043] The training feedback generation unit (150) can generate training feedback data corresponding to the training content and transmit the training feedback data to the user terminal when the virtual content is terminated.

[0044] For example, if the virtual part image of the valve is controlled to a completely released image through the virtual training control unit (140), it is determined that the training content has ended, and the motion of the work tool recognized during the process of releasing the valve can be analyzed to generate the training feedback data.

[0046] More specifically, as illustrated in FIG. 2, the content input unit (120) may include a content calling unit (121) that calls a list of available training content from a designated learning content DB corresponding to the work tool paired with the wearable device, a content list providing unit (122) that transmits the list of available training content called by the content calling unit (121) to the user terminal, and a content setting unit (123) that receives at least one of the training content included in the list of available training content from the user terminal.

[0048] Additionally, as illustrated in FIG. 3, the virtual training control unit (140) may include a motion recognition unit (141) that recognizes the motion of the work tool captured through the camera module provided in the wearable device, and an image control unit (142) that modifies the virtual part image in real time in response to the motion of the work tool recognized by the motion recognition unit (141).

[0049] For example, the image control unit (142) can modify the virtual object image displayed on the wearable device in real time based on the image data set of the virtual object image stored in the learning content DB.

[0050] In addition, the image data set of the virtual object image may include a damaged image of the virtual object image if the motion of the work tool is not performed accurately.

[0051] Accordingly, the training feedback generation unit (150) determines that the training content has ended even when the virtual object image is displayed as a damaged image on the wearable device, and may transmit a notification signal to the user terminal requesting the re-execution of the training content along with the training feedback data.

[0053] Meanwhile, the virtual object display unit (130) can display a work guide image included in the training content by overlaying it on the virtual part image.

[0054] For example, the above work guide image may include a work guide image including the mounting position, mounting direction, etc. of the work tool to mount the work tool to the valve during the process of releasing the valve, and a work guide image including the direction to rotate or move the work tool to release the valve.

[0055] As another example, the above-mentioned work guide image may include text information, and the display method of the work guide image to be displayed on the wearable device through the user terminal may be set.

[0056] Accordingly, the task trainee can perform task training more intuitively by manipulating the task tool on the virtual object image based on the task guide image displayed on the wearable device.

[0058] Meanwhile, the training feedback generation unit (150) generates training feedback data based on the virtual part image at the time when the training content ends and the motion of the work tool captured through the camera module provided in the wearable device, wherein the training feedback data may include the time required for the work, the accuracy of the work, and motion feedback information of the work tool.

[0059] For example, even if the virtual part image at the time when the training content ends is an image of a case where the work is successfully completed, the motion feedback information can be generated based on the motion of the work tool recognized during the work process.

[0060] In addition, the above-mentioned work accuracy can be calculated by comparing the motion learning images stored in the learning content DB with the motion of the work tool to calculate a match rate, and the above-mentioned work accuracy can be calculated based on the match rate.

[0062] As another example, the XR training system (100) utilizing the interaction between the real object and the virtual object may further include a fault diagnosis unit (not shown) that extracts at least one sampled image data from image data collected from the camera module provided in the wearable device and analyzes the sampled image data to diagnose whether the camera module provided in the wearable device is faulty.

[0063] At this time, the fault diagnosis unit (not shown) can calculate an image sensing value corresponding to the sampling image data based on the pixel distribution of the sampling image data extracted from the image data.

[0064] Accordingly, the fault diagnosis unit (not shown) can analyze the image sensing value corresponding to the sampling image data in real time and determine that a fault has occurred in the sensor if the instantaneous rate of change of the image sensing value is greater than or equal to a preset value.

[0065] To this end, in the first embodiment, the instantaneous rate of change (R) of the image sensing value change Calculate ) and the above instantaneous rate of change (R change ) is a preset limit value (S err If it is greater than ), it can be determined that an error has occurred in the image data collected through the camera module (e.g., due to a failure of the camera module itself).

[0066] The instantaneous rate of change (R) of the above image sensing value change ) can be calculated through the derivative f'(x) of the function f(x) representing the characteristics of the image sensor provided in the camera module as shown in [Mathematical Formula 1] below.

[0068] [Mathematical Formula 1]

[0069]

[0070] Here, R change represents the instantaneous rate of change of the image sensing value, and △t represents the time change.

[0072] In the second embodiment, the instantaneous rate of change of the image sensing value calculated in [Equation 1] above is a preset limit value (S err It can be determined that a failure has occurred in the camera module (image sensor) only when the case where the value is greater than ) is greater than a preset number of times during a preset period. In other words, since it is unreasonable to determine that a failure has occurred in the camera module (image sensor) even in cases where the instantaneous rate of change is high in a single instance, a standard can be set so that a failure has occurred in the camera module (image sensor) only when the value is greater than a preset number of times during a preset period in order to improve the accuracy of sensor failure determination.

[0074] In the third embodiment, the instantaneous rate of change of the image sensing value calculated in [Equation 1] above is a preset limit value (S err It can be determined that a failure has occurred in the camera module (image sensor) only when the cumulative number of cases greater than ) is greater than or equal to a preset number. This is because, similar to the second embodiment above, it is unreasonable to determine that the camera module (image sensor) has failed even in cases where the instantaneous rate of change is high in a single instance, and rather than setting a preset period, the criterion is to determine that a failure has occurred in the camera module (image sensor) only when the cumulative number of cases from the time of sensor operation to the present time is greater than or equal to a preset number.

[0076] In the fourth embodiment, if the change in the instantaneous rate of change of the image sensing value calculated in [Equation 1] above occurs rapidly, it can be determined that a failure has occurred in the camera module (image sensor).

[0077] To do this, f'(x) calculated in [Equation 1] is differentiated again to calculate f''(x), and if the value is greater than or equal to a preset value, it can be determined that a failure has occurred in the camera module (image sensor).

[0079] Meanwhile, in the fifth embodiment, even if it is determined that a failure has occurred in the camera module (image sensor) through any one of the first to fourth embodiments, the sensor failure is not immediately confirmed, and the sensor failure may be confirmed when the 'sudden data occurrence condition' described below is simultaneously satisfied.

[0081] Here, the condition for the occurrence of sudden data is to calculate the slopes of the upper line (810, 910) connecting the upper limit values ​​of the image sensing values ​​and the lower line (820, 920) connecting the lower limit values ​​of the image sensing values, as illustrated in FIGS. 4 and 5, and if the difference in slopes calculated in [Equation 2] is greater than or equal to a preset value, it can be determined that sudden data has occurred.

[0082] Accordingly, it is determined that a failure has occurred in the camera module (image sensor) through any one of the first to fourth embodiments above, and at the same time, the slope difference (I) calculated in [Equation 2] diff It may also be set to confirm a sensor failure if ) is greater than or equal to a preset value and satisfies the condition for sudden data occurrence.

[0084] [Mathematical Formula 2]

[0085]

[0086] Here, I max is the slope of the upper limit line connecting the upper limits of the image sensing values, I min is the slope of the lower bound line connecting the lower bounds of the image sensing values, I diff represents the slope difference (absolute value) respectively.

[0088] More specifically, as shown in FIGS. 4 and 5, the image sensing values ​​are collected at regular intervals, and after calculating the slopes of the upper line (810) connecting the upper part of the collected data and the lower line (820) connecting the lower part, the difference in slopes (I) of the two lines diff If the difference is not significant (less than a preset value) when ) is calculated, it is determined that the image sensing value is moving within the error range and is operating normally, and the difference in slope between the two lines (I diff If ) increases beyond a preset value, it can be determined that a malfunction has occurred in the camera module (image sensor) as inaccurate values ​​are being output due to a malfunction in the camera module (image sensor).

[0089] That is, in the case of FIG. 4, since the slope of the upper line (810) is 0.26 and the slope of the lower line (820) is 0.24, the difference in slopes (I) between the two lines diff ) is 0.02, and since this is smaller than the preset reference value of 0.12, it can be determined that the sensor is operating normally.

[0090] Meanwhile, in cases where the upper or lower line appears as a broken line, the slopes can be compared by dividing the area at each bend of the line or by calculating the average of the slopes within a certain period.

[0091] Referring to Fig. 5, we will first explain the case where the slope is compared by dividing the area at each bending section of the straight line. In FIG. 5, if the upper limit line (910) is composed of three lines, and the slope of the first line (911) is 0.26, the slope of the second line (912) is 0.45, the slope of the third line (913) is 0.38, and the slope of the lower limit line (920) is 0.24, then the difference in slope (0.26-0.24) between the first line (911) and the lower limit line (920) is 0.02, which is smaller than the preset reference value of 0.12, so it is judged to be operating normally; however, the difference in slope (0.45-0.24) between the second line (912) and the lower limit line (920) is 0.21, which is larger than the preset reference value of 0.12, so it is judged to be operating abnormally; and in the case of the third line (913), the slope between the lower limit line (920) Since the difference (0.38-0.24) is 0.14, which is greater than the preset threshold value of 0.12, it can be determined that the operation is abnormal. In this case, where the area is divided at each bend of the straight line to compare the slopes and determine whether a failure has occurred, the time at which the failure occurred can also be estimated, making it very useful when the time of failure needs to be identified.

[0092] Next, the slope can be compared by calculating the average slope within a certain period. In the entire section of FIG. 5, the average slope of the upper line (910) is {(0.24+0.45+0.38) / 3}=0.35, and since the difference from the average slope (0.24) of the lower line (920) is 0.11, it is smaller than the preset reference value of 0.12, so it can be determined that it is operating normally. In this way, determining whether there is a failure by calculating the average slope within a certain period does not determine it as a failure even if there are temporary slope fluctuations, as long as the average value does not exceed the reference value. This allows the sensitivity of failure determination to be lowered somewhat, thereby preventing unnecessary failure determinations.

[0094] <Example 2>

[0095] Meanwhile, an XR training method utilizing interaction between a real object and a virtual object according to an embodiment of the present invention may include: a step of pairing the wearable device and the work tool through the work tool pairing unit; a step of receiving at least one training content input through the user terminal when the pairing between the wearable device and the work tool is completed; a step of the virtual object display unit displaying a virtual part image corresponding to the training content received by the content input unit on a display module provided in the wearable device; a step of the virtual training control unit recognizing the motion of the work tool through a camera module provided in the wearable device and controlling the display state of the virtual part image based on the motion of the work tool; and a step of, when the training content is terminated, the training feedback generation unit generating training feedback data corresponding to the training content and transmitting the training feedback data to the user terminal.

[0097] Accordingly, according to the present invention as described above, an XR training system and method utilizing the interaction between a real object and a virtual object can be provided, which can train a work process more effectively without consuming a virtual object provided as a consumable, by recognizing the motion of a work tool paired with a wearable device and training a work process using a virtual object displayed on the wearable device.

[0098] In addition, an XR training system and method utilizing the interaction between real and virtual objects can be provided, which can provide customized work guide information necessary for the work process to the work trainee by analyzing the motion of the work tool recognized through the work process and generating feedback information necessary for the work process and providing it to the work trainee.

[0100] In addition, a control method for an XR training system utilizing the interaction between a real object and a virtual object according to an embodiment of the present invention may be recorded on a computer-readable medium containing program instructions for performing operations implemented by various computers. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions on the medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0102] As described above, although an embodiment of the present invention has been explained by limited embodiments and drawings, the embodiment of the present invention is not limited to the embodiments described above, and various modifications and variations are possible from this description by those skilled in the art to which the present invention pertains. Accordingly, an embodiment of the present invention should be understood only by the claims described below, and all equivalent or analogous variations thereof shall be considered to be within the scope of the inventive concept. Explanation of the symbols

[0104] 110: Work tool pairing part 120 : Content input section 121 : Content call section 121 : Content List Provider 123 : Content Settings 130 : Virtual object display unit 140: Virtual training control unit 141: Motion recognition unit 142 : Image Darebu 150 : Training Feedback Generation Section

Claims

Claim 1 A work tool pairing unit that pairs a preset wearable device with at least one work tool; a content input unit that receives at least one training content through a preset user terminal when the pairing of the wearable device and the work tool is completed through the work tool pairing unit; a virtual object display unit that displays a virtual part image corresponding to the training content inputted in the content input unit on a display module provided in the wearable device; and a virtual training control unit that recognizes the motion of the work tool through a camera module provided in the wearable device and controls the virtual part image based on the motion of the work tool. and, when the training content is terminated, a training feedback generation unit that generates training feedback data corresponding to the training content and transmits the training feedback data to the user terminal; further includes a fault diagnosis unit that extracts at least one sampling image data from image data collected from the camera module provided in the wearable device and analyzes the sampling image data to diagnose whether the camera module provided in the wearable device is faulty; the content input unit includes a content calling unit that calls a list of available training content corresponding to the work tool paired with the wearable device from a pre-designated learning content DB; a content list providing unit that transmits the list of available training content called by the content calling unit to the user terminal; and a content setting unit that receives at least one of the training content included in the list of available training content from the user terminal; and the virtual training control unit includes a motion recognition unit that recognizes the motion of the work tool captured through the camera module provided in the wearable device.and an image control unit that modifies the virtual part image in real time in response to the motion of the work tool recognized by the motion recognition unit; wherein the image control unit modifies the virtual object image displayed on the wearable device in real time based on an image data set of virtual object images stored in the learning content DB, and the image data set of virtual object images includes an image of a damaged virtual object image in response to cases where the motion of the work tool is not performed accurately, and the virtual object display unit displays a work guide image included in the training content by superimposing it on the virtual part image, wherein the work guide image includes image data guiding at least one of the mounting position, mounting direction, rotation direction, and movement direction of at least one of the work tool and work part, and text data corresponding to the image data, and the training feedback generation unit generates training feedback data based on the virtual part image at the time when the training content ends and the motion of the work tool captured through the camera module provided in the wearable device, wherein the training feedback data includes work time, work accuracy, and motion feedback information of the work tool, and the failure The diagnostic unit calculates an image sensing value corresponding to the sampled image data based on the pixel distribution of the sampled image data extracted from the image data, analyzes it in real time, and calculates the instantaneous rate of change (R) of the image sensing value calculated based on the following [Equation 1]; change ) is the first limit value (S) that is pre-set err A first failure condition in which it is determined that an error has occurred in the image data due to a failure in the image sensor provided in the camera module when it is greater than ); the instantaneous rate of change (R) of the image sensing value change ) is the above limit value (S err A second failure condition in which a failure is determined to have occurred in the image sensor only when the case where it is greater than ) is greater than or equal to a preset number of times during a preset period; and the instantaneous rate of change (R) of the image sensing value change ) is a preset limit value (S err A third failure condition in which a failure is determined to have occurred in the image sensor only when the cumulative number of cases greater than ) is greater than or equal to a preset number; the instantaneous rate of change (R) of the image sensing value change A fourth failure condition in which f'(x) calculated in the following [Equation 1] is differentiated again to calculate f''(x) in order to detect cases where a sudden change occurs, and if said f''(x) is greater than a preset second limit value, it is determined that a failure has occurred in the image sensor; and a fifth failure condition in which, even if it is determined that a failure has occurred in the image sensor through at least one of the first to fourth failure conditions, the failure is not immediately confirmed, but the failure of the image sensor is confirmed only when the condition for sudden data occurrence is simultaneously satisfied; applying any one of these to determine whether a failure has occurred in the camera module, and [Equation 1] (Here, R change (where represents the instantaneous rate of change of the image sensing value and △t represents the time change) The above sudden data occurrence condition calculates the slopes of the upper line connecting the upper limits of the image sensing values ​​and the lower line connecting the lower limits of the image sensing values, respectively, and the slope difference (I) calculated in [Mathematical Formula 2] below diff An XR training system utilizing the interaction between real and virtual objects, characterized by determining that sudden data has occurred when ) is greater than a preset third threshold value.[Equation 2] (Here, I max is the slope of the upper limit line connecting the upper limits of the image sensing values, I min is the slope of the lower bound line connecting the lower bounds of the image sensing values, I diff ) Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete

Images