Training system based on mixed reality

Abstract

A mixed reality-based training system for building inspection training comprises: a projection module for projecting one or more virtual images associated with an augmented reality environment linked to a real-world environment of a building for inspection by an inspector; a user input sensing module for capturing user input related to interaction between the real-world environment and the inspector during building inspection training; and a training evaluation module to determine one or more metrics associated with a training progress of the inspector based on the user input captured during a building inspection training.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a mixed reality based training system, in particular, but not exclusively, to a mixed reality based training system for building inspection training. BACKGROUND

[0002] Newly built building units need to be inspected and maintained. As the number of newly built units increases due to urban expansion, and as diversified designs are adopted to meet the increasingly complex demands of the market, the demand and workload of building inspection also increase. Building management needs to adapt to diversified inspection work. SUMMARY

[0003] According to a first aspect of the present disclosure, there is provided a mixed reality based training system for building inspection training. The mixed reality based training system comprises:

[0004] a projection module configured to project one or more virtual images associated with an augmented reality environment linked to a real world environment of a building for inspection by an inspector;

[0005] a user input sensing module configured to capture user inputs related to interactions between the real world environment and the inspector during building inspection training; and

[0006] a training evaluation module configured to determine one or more indicators associated with a training progress of the inspector based on the user inputs captured during building inspection training.

[0007] According to the first aspect, the projection module is configured to trigger an update of the augmented reality environment in response to the interactions between the virtual images and the inspector.

[0008] According to the first aspect, the mixed reality based training system further comprises a hand tracking module configured to capture motion data associated with hand movements of the inspector during inspection.

[0009] According to the first aspect, the captured motion data is compared with one or more preset parameters, each of which is related to an interaction with the virtual images, thereby simulating interactions between the real world inspection of the building and the inspector.

[0010] According to the first aspect, the projection module is further configured to visually simulate a virtual instructor configured to visually project guidance associated with the inspection of the augmented reality environment to the inspector.

[0011] According to the first aspect, the projection module is configured to project a first image associated with a virtual building unit, and the training evaluation module is configured to determine the indicator associated with the training progress of the inspector based on a comparison between the projected first image and a second image captured by the inspector.

[0012] According to the first aspect, the projection module is configured to visually simulate information associated with building quality and inspection test standards to the inspector.

[0013] According to the first aspect, the projection module is configured to generate a quiz associated with the real-world inspection of the building graphically represented on the augmented reality environment, and the training evaluation module is configured to evaluate the user input associated with an answer of the quiz based on user input captured by the user input sensing module.

[0014] According to the first aspect, the training evaluation module is configured to store personal information, training time, and training evaluation records of the inspector.

[0015] According to the first aspect, the training evaluation module is configured to feed the indicators associated with the training progress of a plurality of inspectors into an AI model, and evaluate training performance of an individual inspector relative to the plurality of inspectors. BRIEF DESCRIPTION OF DRAWINGS

[0016] Embodiments of the present disclosure will be described below by way of example with reference to the accompanying drawings.

[0017] Figure 1 A block diagram of a mixed reality based training system according to an embodiment of the present disclosure.

[0018] Figure 2 A schematic diagram of a first mixed reality interface according to an embodiment of the present disclosure.

[0019] Figure 3 A schematic diagram of an architecture and distinguishing features of a mixed reality based training system according to an embodiment of the present disclosure.

[0020] Figure 4 A schematic diagram of a second mixed reality interface according to an embodiment of the present disclosure.

[0021] Figure 5 A schematic diagram of a third mixed reality interface according to an embodiment of the present disclosure.

[0022] Figure 6 A schematic diagram of a fourth mixed reality interface according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Currently, property management relies on experienced inspectors to train new employees, but this traditional approach has limited capacity and is labor-intensive. Due to the shortage of manpower, it is difficult to meet the needs of property maintenance.

[0024] The present disclosure relates to a training system that combines digital twin with mixed reality (MR) technology. Specifically, digital twin can be used to simulate or emulate the operation of physical equipment in a virtual world. Mixed reality can also be used to combine virtual and real elements in a coordinated and interactive manner. For example, a digital twin can demonstrate structural components of a system that are displayed through a user interface and are computer-generated, which can be combined with a real environment.

[0025] Reference Figure 1 An embodiment of a mixed reality-based training system 100 for building inspection training is shown in FIG. 1. The mixed reality-based training system 100 includes a projection module 110 for projecting one or more virtual images associated with an augmented reality environment 220 linked to a real-world environment 12 of a building 10 for inspection by an inspector 20, a user input sensing module 111 for capturing user inputs related to interactions between the real-world environment 12 and the inspector 20 during building inspection training, and a training evaluation module 140 for determining one or more indicators associated with a training progress of the inspector 20 based on the user inputs captured during the building inspection training.

[0026] Generally, a building 10 is constructed according to submitted design drawings, and when the construction is almost complete, the construction site should be inspected by an inspector 20 to identify any potential defects, irregularities, and violations that can pose a potential danger to public safety or do not comply with relevant regulatory requirements. The inspector 20 must inspect the wall paint, fire safety, drainage system, power supply system, etc. to ensure everything is normal, and all problems have been rectified before the construction project is considered complete and the occupants move in. For the purposes of this document, the term “building” includes all types of properties, including indoor and outdoor sites, such as but not limited to apartments, houses, factories, warehouses, skyscrapers, stores, restaurants, hotels, and power plants. The term “building” can also include other types of tangible assets, such as ships, airplanes, submarines, and cargo.

[0027] As Figure 1As shown, a schematic view of a mixed-reality based training system 100 and a building 10 to be inspected by an inspector 20 is shown. The mixed-reality based training system 100 can be embodied as an accessory, for example, a head mounted display (HMD) embedded with a computing device. Alternatively, the virtual-reality based training system 100 can also be part of a computing device in signal communication with a HMD or an LCD screen.

[0028] Reference is made to Figure 1 Further details are obtained to achieve the overall architecture of the mixed-reality based training system 100 in accordance with an exemplary embodiment of the present disclosure. In essence, the mixed-reality based training system 100 includes a projection module 110 for simulating an augmented reality environment overlaying a real-world environment 12 during building inspection training. While the present disclosure relates to mixed-reality, it is also possible to simulate the real-world environment 12 as a virtual reality environment on an additional projector, whereby the augmented reality environment 220 can overlay such simulated virtual reality environment to achieve similar effects.

[0029] The present disclosure utilizes MR HMD features in the context of building inspection training. For example, the projection module 110 can be part of a MR headset, and more preferably, part of a head mounted display (HMD), and displays a plurality of multimedia content to the inspector 20. The head mounted display (HMD) can be a Microsoft HoloLens comprising smart glasses, and the multimedia content is projected on the smart glasses to the inspector 20. The realistic guided vision inside the MR headset can provide an immersive experience to the user, enabling him to undergo building inspection training as if he is being trained by an experienced inspector in a face-to-face training environment.

[0030] An image capturing module 120 is also provided for capturing images of the construction site during inspection. For example, the image capturing module 120 is operable to capture images of objects at a distance relative to the inspector 20. Subsequently, these signals are processed by a performance evaluation module 140. Preferably, these modules are embedded within the head mounted display (HMD), and are movable with the inspector 20.

[0031] Preferably, the mixed-reality based training system 100 also includes one or more user input sensing modules for detecting user inputs related to interactions between the real-world environment 12 and the inspector 20, so as to monitor the activities of the inspector 20 and other operations associated with building inspection of the building 10.

[0032] In one exemplary embodiment, the user input sensing module may further include a hand tracking module 130, which is used to capture motion data associated with the inspector 20's hand movements during a building inspection. Specifically, the inspector 20 performs hand tracking to identify interactions between the multimedia content displayed on the projector 110 and the inspector 20, thereby triggering updates to the augmented reality environment. For example, the inspector 20's hand movements may be categorized as tapping or photo-capturing actions to trigger further actions. This allows the inspector 20 to provide local control, maximizing the sense of presence and realism during the building inspection.

[0033] Preferably, the hand tracking module 130 enables the inspector 20 to interact with the virtual environment through natural hand gestures, without the need for a physical controller. Optionally, the head-mounted display (HMD) may also provide multiple physical buttons that the inspector 20 can press to select or deselect one or more functions associated with the virtual buttons.

[0034] exist Figure 1 In an alternative embodiment not shown, the user input sensing module may further include an eye-tracking module for capturing gaze data related to the direction of the inspector 20's gaze during an inspection of building 10. For example, the inspector 20's eyes may gaze toward a virtual button projected onto projection module 110, and a double blink may indicate confirmation of selecting a function associated with the gazed virtual button. Therefore, eye tracking can accurately monitor and analyze the user's eye movements in a mixed reality environment to enhance the immersion and usability of the MR experience.

[0035] In one exemplary embodiment, the training evaluation module 140 is used to determine one or more metrics related to the training progress of inspector 20 during building inspection training, based on captured motion data. For example, the training evaluation module 140 may determine the training progress of inspector 20 based on the learning time of inspector 20 and the evaluation results of tests attempted by inspector 20. Furthermore, the training evaluation module 140 may also determine the ranking of inspector 20 within the group of inspectors 20 trained through the mixed reality-based training system 100.

[0036] In an example embodiment, the training evaluation module 140 is part of a computing module 150 that includes suitable components necessary to receive, store, and execute appropriate computer instructions. These components can include a processing unit 161 that includes a central processing unit (CPU), a math co-processing unit (Math Processor), a graphics processing unit (GPU), or a tensor processing unit (TPU) for tensor or multi-dimensional array computation or manipulation operations; a read-only memory (ROM) 162; a random-access memory (RAM) 163; input / output (I / O) devices, such as disk drives 164; and a user interface 165, such as a keyboard, touch screen, or digital instrument panel. The processing unit 161 can be a single processor to provide the combined functionality of multiple processors. In this example embodiment, the computing module 150 is used to receive data associated with the real-world environment 12 and the inspector 20, as well as data measured by external sensing units, such as the image capture module 120 and the hand tracking module 130.

[0037] The computing module 150 can also include other input devices, such as Ethernet ports, USB ports, and the like; a display 166, such as a liquid crystal display, a light emitting display, or any other suitable display; and a communication link (i.e., a communication interface) 170.

[0038] The computing module 150 can include instructions that can be stored in the ROM 162, the RAM 163, or the disk drives 164 and executed by the processing unit 161. One or more communication interfaces (i.e., one or more communication links) can be provided that can connect one or more computing devices, such as servers, personal computers, terminals, wireless or handheld computing devices, Internet of Things devices, smart devices, and edge computing devices, in different ways. At least one of the multiple communication links can be connected to an external computing network through a telephone line or other type of communication link. The communication interface is configured to allow data communication over any suitable communication network using any suitable protocol (e.g., Wi-Fi, Bluetooth, 4G, 5G, or any other suitable protocol).

[0039] The computing module 150 can also provide the necessary computing power to operate or interface with machine learning networks, such as neural networks, to provide various functions and outputs, particularly for the training and testing of the multi-modal decision algorithm. The neural networks can be implemented locally, or can also be accessed or partially accessed through a server or cloud-based service. The machine learning networks can also be untrained, partially trained, or fully trained, and / or can also be retrained, modified, or updated over time. The computing module 150 can include the computing power to execute the simulation system. The computing module 150 can also include the computing power to execute the algorithm with several layers.

[0040] Preferably, the mixed-reality based training system 100 can also include one or more speakers 115 for generating a plurality of audible outputs related to the multimedia content projected by the projection module 110 during the building inspection training. More preferably, the speakers 115 can include a pair of speaker units for providing different audible outputs in left and right channels, respectively. Alternatively, the speakers 115 can be integrally formed as part of a head-mounted display (HMD), or as an external device in signal communication with the computing module 150. The speakers 115 can also generate one or more sound outputs associated with the video guidance or quiz.

[0041] The mixed-reality based training system 100 can also include one or more databases for storing different data utilized and processed by the processor 161. For example, the mixed-reality based training system 100 can include a cloud server database 180 for storing instructions for generating the augmented reality environment. The cloud server database 180 can store pre-recorded guidance videos by experienced inspectors. The cloud server database 180 can store personal information related to the trainee inspector 20, such as the training time and training assessment records of the inspector 20. The cloud server database 180 can also store building quality and inspection test standards, and versions of the test standards can be updated periodically.

[0042] For example, the computing module 150 is configured to transmit one or more signals to the projection module 110 to display one or more images associated with the augmented reality environment linked to the real-world environment 12 to the inspector 20. To construct the augmented reality environment, the computing module 150 can retrieve real data from the cloud server database 180 and generate signals associated with the augmented reality environment. The computing module 150 can transmit one or more signals to the projection module 110 to display virtual images associated with the augmented reality environment to the inspector 20. The computing module 150 then receives one or more signals from the image capture module 120 and the hand tracking module 130; the processing unit 161 processes the captured data to determine the training performance of the inspector 20.

[0043] Preferably, the computing module 150 can be preloaded with one or more instructions to execute one or more virtual buttons and unique features in order to better utilize the augmented reality environment and provide advanced features beyond the training session. For example, these instructions can be executed by the processing unit 161 to implement various different features such as gesture simulation, video guidance, photo comparison, building quality and inspection test standards, question and answer, retrieval of trainee personal information, training time and training evaluation records, and AI data evaluation.

[0044] Figure 2 An exemplary mixed reality interface 200 according to the present disclosure is shown, which is a combination of a real-world environment 210 (e.g., an indoor environment of a building 10 to be inspected by a trainee building inspector 20) and an augmented reality environment in the form of a virtual interface 220. The virtual interface 220 can be projected to the trainee building inspector 20 as a start-up screen on the projection module 110.

[0045] In this embodiment, the virtual interface 220 can be a home screen that provides a main menu with a plurality of interactable virtual buttons 222, 224, 226, 228, 230, each corresponding to a particular function that can be selected by the trainee building inspector 20 to select a corresponding function. Upon receiving a user input, a user interaction interface will be rendered on the screen of the projection module 110. For example, the user input can be communicated through the user’s gaze direction or gesture movement.

[0046] The video guidance button 222 allows the trainee building inspector 20 to call up a guidance start-up screen that can provide some guidance for the inspection of various building units within the building inspection site. For example, one or more pre-recorded instructional videos on different topics are stored on the cloud server database 180 and can be called up and projected onto the projection module 110. This can simulate a virtual training by a virtual instructor.

[0047] The photo comparison button 224 allows the trainee building inspector 20 to call up a photo comparison start-up screen that can compare a user input with a pre-stored reference photo stored on the cloud server database 180. For example, the trainee building inspector 20 can be required to follow instructions and attempt to identify some defects in the building 10. The trainee building inspector 20 can capture some defects with the image capture module 120 and then the computing module 150 can compare the similarity between the reference photo and the image captured by the trainee building inspector 20.

[0048] The building quality and inspection test standards button 226 allows the trained building inspector 20 to access a building and inspection standards startup screen, which provides a manual on the test standards on the screen of the projection module 110. The content of the standards can also be presented to the trained building inspector 20 audibly via the speaker 115.

[0049] The test button 228 allows the trainee building inspector 20 to access a test launch screen, which can present questions to test the trainee's knowledge. Alternatively, the test button 228 can also trigger an evaluation mode when the trainee building inspector 20 enters the building 10 to be inspected. The training progress of the trainee building inspector 20 can be evaluated through the inspection of the real-world environment 210 and the completion of tasks.

[0050] The trainee profile button 230 allows the trainee building inspector 20 to access a trainee profile interface, which can display one or more pieces of information related to the trainee building inspector 20 on the screen of the projection module 110. For example, the trainee building inspector 20 can view his / her accumulated training time, such as completed video tutorials, reading time spent on building and inspection standards, and other tasks assigned by the training system 100.

[0051] In one exemplary embodiment, such as Figure 2 The interactive virtual buttons 222, 224, 226, 228, and 230 shown are similar to... Figure 3 The present disclosure relates to several distinguishing features 300. Distinguishing features 300 include gesture simulation function 310, AI data evaluation 320, training time 330, and training evaluation record 340. These distinguishing features 300 can extend the application of building inspection training to a wider range, which will be discussed in conjunction with... Figure 3 Describe it.

[0052] The gesture simulation function 310 can associate one or more gestures (e.g., a user's hand grasping or poking motion) with actions. For example, a poking gesture will simulate object selection in the real-world environment 210, while a two-handed photo-taking gesture will simulate an instruction to take a photo of the real-world environment 210 via the image capture device 114.

[0053] AI Data Evaluation Function 320 can perform some data processing with the help of one or more deep learning networks (such as neural networks) to perform some statistical collection and analysis.

[0054] The training time function 330 can store the cumulative training time spent by the trainee building inspector 20 on video-guided training and the time spent by the trainee building inspector 20 on reading the building quality and inspection test standards. For example, the system 100 can require each trainee building inspector 20 to undergo a minimum amount of unsupervised training before he or she can take the quiz test.

[0055] The training assessment record function 340 can also store the previous training results of the quiz test attempted by the trainee building inspector 20. The training assessment record function 340 can further divide the training topics into specific areas and indicate to the trainee building inspector 20 the training performance in each area and his or her strengths and weaknesses.

[0056] With reference to FIG. 4, Figure 4 The mixed reality interface 400 is a combination of the real-world environment 410 and a virtual interface projected on the screen of the projection module 110 when the video guidance button 222 is triggered by the trainee building inspector 20. The virtual interface can be projected to the trainee building inspector 20 as a start-up screen 420 on the projection module 110 and overlaid on the real-world environment 410. The start-up screen 420 can provide a plurality of interactive virtual buttons 422, 424, 426, 428, 430, 432, each corresponding to an inspection perspective of a respective tutorial that can be selected by the trainee building inspector 20.

[0057] In the building 10, there can be some common types of building defects that can be visually inspected by the inspector 20. For example, this includes non-structural cracks, which usually appear between non-structural walls and structural members and extend in many directions. Worse still, there can also be some structural cracks that will penetrate the surface rendering into structural concrete or load-bearing brick walls. Spalling of concrete can also be shown by water stains or rust marks on the surface of the concrete chamber.

[0058] In addition to building defects, drainage system failures are also a common defect that can exist in various forms, whether in new or old buildings 10. The most common are internal drain pipes rusting and external wall drain pipes leaking or breaking. In some cases, workers can have incorrectly connected the pipes and inadvertently changed the drainage system.

[0059] In this embodiment, the defect measurement button 422 can prompt the playing of a video to guide the trainee building inspector 20 on how to correctly measure the size of a crack. The water leakage button 424 can prompt the playing of a video to guide the trainee building inspector 20 on how to accurately record water leakage. The wall surface button 426 can also prompt the playing of a video to guide the trainee building inspector 20 on the inspection of wall surfaces.

[0060] Fire safety is also a major topic of building design and inspection. For example, some building components are designed to be fire resistant and to prevent the spread of fire, thereby protecting the occupants and their property from the ravages of fire. It is extremely important to provide adequate means of escape in the event of fire and other emergencies.

[0061] In this embodiment, the fire safety button 428 can prompt the playing of a video to instruct on a fire safety inspection checklist. For example, fire equipment should be tested and the expiration date of the fire equipment (e.g., fire extinguisher) should be checked.

[0062] In this embodiment, the hazard prevention button 430 can prompt the playing of a video to instruct the trainee building inspector 20 to properly display emergency notices along the hallways and indoor walls of the building 10. For example, the trainee building inspector 20 should check whether emergency exit signs in the building 10 are mislocated or missing. The trainee building inspector 20 should also check whether there are any swing barriers to doors, obstructed escape routes, or other blocked escape routes.

[0063] Finally, there is also provided a lighting button 432, which can prompt the playing of a video to instruct the trainee building inspector 20 to conduct a light fixture inspection. For example, the trainee building inspector 20 should check whether public areas have adequate lighting. The trainee building inspector 20 should also periodically check the emergency lighting system.

[0064] Reference Figure 5 FIG. 5 shows an example mixed reality interface 500 according to the present disclosure, which is a combination of a real-world environment 510 (e.g., an indoor environment of a building site to be inspected by the trainee building inspector 20) and a virtual interface 520. The virtual interface 520 is projected onto the projection module 110 as a launch interface when the trainee building inspector 20 triggers the quiz button 228.

[0065] This indoor area is a closed space 522 through which occupants normally pass through a doorway 524. In this space 522, there are a number of inspection points that need to be checked by an experienced building inspector. For example, the escape route of the doorway 524 is obstructed by some unpacked cartons 526. A lighting fixture 528 has been installed and hung vertically from the ceiling, but the light bulb has not been installed. While the emergency lighting 530 has been properly affixed to the wall and is in close proximity to the doorway 524, this lighting is not connected to a power source and cannot provide a clear indication of escape in the event of a power outage. There are also some cracks 532 in the wall that extend along the interior wall, and there is a water leak 534 on the pipe above the power outlet 536. A fire extinguisher 538 is also placed in a hidden location behind the door 524, which is not in compliance with fire safety regulations. All of these issues would normally be discovered by an experienced building inspector.

[0066] During the testing session, the trainee building inspector 20 is required to perform necessary actions according to the standard inspection procedure. For example, the trainee building inspector 20 is required to find as many problems as possible to pass the test. To detect the selection of objects in the real-world environment 510, the trainee building inspector 20 can point at the corresponding objects, and the hand tracking module 130 will capture the pointing action to the objects, respectively.

[0067] The trainee building inspector 20 is also required to take some photos of the relevant building areas to complete the procedure and prepare the inspection report. For example, the trainee building inspector 20 can make a photo-taking gesture, and the hand tracking module 130 will recognize the instruction and instruct the image capturing module 120 to take the image of the real-world environment 510.

[0068] Finally, the detailed workflow of the testing inspection is shown with reference to Figure 6 , the trainee building inspector 20 is required to walk around the building site and inspect all the hazards and building defects. The trainee building inspector 20’s performance in the test will be evaluated by the performance evaluation module 150 according to one or more user inputs of the trainee building inspector 20.

[0069] First, the trainee building inspector 20 is required to inspect the hazards. The trainee building inspector 20 is required to notice the presence of the obstacle 526 and remove the unpacked carton 526 from the escape route. The performance evaluation module 150 will determine whether the unpacked carton 526 is removed accordingly by the image capturing device 120.

[0070] The trainee building inspector 20 is also required to test the performance of the fire extinguisher 538 and take a photo of the product label on the fire extinguisher 538 to identify the expiration date and record. Finally, the trainee building inspector 20 is required to move the fire extinguisher 538 to a more visible location 610 to comply with the fire safety regulations.

[0071] Next, the trainee building inspector 20 is required to inspect the function of the lighting device. For example, the trainee building inspector 20 is required to turn off the main power supply and test whether the emergency power supply can power the emergency lighting 530. The trainee building inspector 20 is also required to test the lighting device 528 and replace the light source 620 (e.g., light bulb) to the lighting device 528.

[0072] Finally, the trainee building inspector 20 is also required to inspect the defects of the building and drainage system. The trainee building inspector 20 is required to take a photo of the crack 532 and measure the size of the crack 532 according to the inspection standard. The trainee building inspector 20 is also required to take a photo of the water leakage of the water pipe 534 and immediately turn off the main water source to avoid potential water damage.

[0073] Preferably, the present disclosure applies mixed reality smart glasses technology and cooperates with the property management industry to provide high-quality and efficient training courses for new employees in the industry. Therefore, it can provide comprehensive and intuitive interactive exercises for trainees, combined with the guidance content of experienced experts, to improve the quality and process of building inspection training.

[0074] Preferably, the present disclosure can also be customized according to the progress of the trainees and the needs and designated buildings of building inspectors or related parties. By integrating AI analysis of training results, the present disclosure can evaluate the training performance of each trainee.

[0075] Advantageously, the present disclosure develops a mixed reality (MR) driven inspection process that can realize the digitization, standardization and intelligentization of building inspection and verification processes. Therefore, the present disclosure can significantly improve the inspection efficiency and quality.

[0076] While not required, embodiments described in reference to the drawings can be implemented as an application programming interface (API) or a series of libraries used by developers, or can be included in another software application, such as a terminal or personal computer operating system or a portable computing device operating system. Generally, as program modules include routines, programs, objects, components, and data files that help perform particular functions, those skilled in the art will appreciate that the functions of the software application can be distributed among multiple routines, objects, or components to achieve the same functionality as required herein.

[0077] It should also be understood that where the methods and systems of the present disclosure are implemented in whole or in part by a computing system, any appropriate computing system architecture can be used. Computing systems can include tablet computers, wearable devices, smartphones, Internet of Things devices, edge computing devices, standalone computers, network computers, cloud-based computing devices, and special-purpose hardware devices. Where the terms "computing system" and "computing device" are used, these terms are intended to encompass any appropriate configuration of computer hardware capable of implementing the described functionality.

[0078] Those skilled in the art will understand that various changes and / or modifications can be made to the present disclosure as shown in the specific embodiments without departing from the spirit or scope of the present disclosure as broadly described. Therefore, the above embodiments are considered illustrative in all respects and are not restrictive.

[0079] Any reference contained herein to prior art is not to be taken as an admission that the information is common general knowledge.

Claims

1. A mixed reality based training system for building inspection training, characterized in that, comprising: a projection module for projecting one or more virtual images associated with an augmented reality environment; the augmented reality environment is linked to a real-world environment of a building for inspection by an inspector; a user input sensing module for capturing user inputs related to interactions between the real-world environment and the inspector during building inspection training; and a training evaluation module for determining one or more indicators associated with a training progress of the inspector based on the user inputs captured during building inspection training.

2. The mixed-reality based training system of claim 1, wherein, wherein, the projection module is for triggering updates to the augmented reality environment in response to the interactions between the virtual images and the inspector.

3. The mixed-reality based training system of claim 2, wherein, further comprising: a hand tracking module for capturing motion data associated with hand movements of the inspector during inspection.

4. The mixed-reality based training system of claim 3, wherein, wherein, the captured motion data is compared with one or more preset parameters, each of the preset parameters being related to interactions with the virtual images, thereby simulating interactions between the real-world inspection of the building and the inspector.

5. The mixed reality-based training system of claim 1, wherein, wherein, the projection module is further for visually simulating a virtual instructor for visually projecting guidance to the inspector associated with the inspection of the augmented reality environment.

6. The mixed-reality based training system of claim 1, wherein, wherein, the projection module is for projecting first images associated with virtual building units, and the training evaluation module is for determining the indicators associated with the training progress of the inspector based on a comparison between the projected first images and second images captured by the inspector.

7. The mixed reality-based training system of claim 1, wherein, wherein, the projection module is for visually simulating information associated with building quality and inspection test standards to the inspector.

8. The mixed reality-based training system of claim 1, wherein, wherein, the projection module is for generating a quiz associated with a real-world inspection of the building graphically represented on the augmented reality environment, and the training evaluation module is for evaluating the user inputs associated with answers to the quiz based on user inputs captured by the user input sensing module.

9. The mixed reality-based training system of claim 1, wherein, wherein, the training evaluation module is for storing personal information, training time, and training evaluation records of the inspector.

10. The mixed-reality based training system of claim 1, wherein, wherein, the training evaluation module is for feeding the indicators associated with the training progress of a plurality of inspectors into an AI model, and evaluating training performance of individual inspectors relative to the plurality of inspectors.

Images