Training method, device, equipment and system based on augmented reality equipment

By superimposing digital twin models and real scenarios on extended real-life equipment, independent maintenance training for central air conditioners and other equipment is achieved, the problems of high training costs and difficult to monitor in the existing technology are solved, and the training efficiency and effect are improved.

CN119091090APending Publication Date: 2024-12-06SHANGHAI MEICON INTELLIGENT CONSTR CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310658734.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In the prior art, maintenance training for large HVAC equipment such as central air conditioners requires guidance from professional engineers, which leads to high training costs and takes up tutor time, making it difficult to effectively monitor and optimize the training effect.

Method used

Using a training method based on extended reality equipment, we superimpose the digital twin models and real scenarios of the equipment to be repaired, and obtain guidance information in response to interactive operations, realize independent learning and maintenance training, and reduce the direct participation of the instructor.

Benefits of technology

It reduces training costs, improves training efficiency and effectiveness, and can monitor and optimize training plans in real time to meet personalized training needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091090A_ABST
    Figure CN119091090A_ABST
Patent Text Reader

Abstract

The invention discloses a training method based on augmented reality equipment, a training device based on the augmented reality equipment, the augmented reality equipment, a training system, a nonvolatile computer readable storage medium and a computer program product. The training method comprises the steps that according to position information of to-be-maintained equipment and augmented reality equipment, a digital twin model of the to-be-maintained equipment and the to-be-maintained equipment are displayed in an overlapped mode; and in response to an interactive operation on the digital twin model, obtaining guide information of the to-be-maintained equipment, the guide information at least comprising one or more guide steps for maintaining the to-be-maintained equipment. According to the method, the to-be-maintained equipment and the digital twin model are displayed in an overlapping manner, and then the guide information of the to-be-maintained equipment is acquired in response to the interactive operation on the digital twin model. And then the to-be-trained object learns based on the guiding steps in the guiding information, and the to-be-maintained equipment is maintained to complete maintenance training, so that training based on the augmented reality equipment is completed, and the teaching cost is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of equipment training, and more specifically, to a training method based on an extended reality device, a training apparatus based on an extended reality device, an extended reality device, a training system, a non-volatile computer-readable storage medium, and a computer program product. Background Art

[0002] In the prior art, for large HVAC equipment such as central air conditioners, maintenance engineers usually perform on-site troubleshooting and other operations, which requires high professional skills of maintenance engineers. Therefore, the training of maintenance engineers is very important. However, the existing training is basically conducted by engineers with strong professional capabilities as training instructors, who train the trainees. This will take up a lot of time and energy of the training instructors and delay their normal work, resulting in high teaching costs for the trainees. Summary of the invention

[0003] Embodiments of the present application provide a training method based on an extended reality device, a training apparatus based on an extended reality device, an extended reality device, a training system, a non-volatile computer-readable storage medium, and a computer program product.

[0004] The training method based on extended reality device of the implementation mode of the present application includes superimposing and displaying the digital twin model of the equipment to be repaired and the equipment to be repaired according to the location information of the equipment to be repaired and the extended reality device; in response to the interactive operation of the digital twin model, obtaining the guidance information of the equipment to be repaired, and the guidance information at least includes one or more guidance steps for repairing the equipment to be repaired.

[0005] The training device based on the extended reality device of the embodiment of the present application includes an overlay module and an acquisition module. The overlay module is used to overlay and display the digital twin model of the device to be repaired and the device to be repaired according to the location information of the device to be repaired and the extended reality device; the acquisition module is used to obtain the guidance information of the device to be repaired in response to the interactive operation of the digital twin model, and the guidance information at least includes one or more guidance steps for repairing the device to be repaired.

[0006] The extended reality device of an embodiment of the present application includes a display, one or more processors, a memory and one or more computer programs, wherein one or more of the computer programs are stored in the memory and executed by one or more of the processors, and the computer program includes instructions for executing the training method of any of the above embodiments.

[0007] The management system for equipment to be repaired according to the implementation mode of the present application includes equipment to be repaired and an extended reality device, wherein the extended reality device is communicatively connected with the equipment to be repaired, and the extended reality device is used to execute the training method described in any one of the above implementation modes.

[0008] The non-volatile computer-readable storage medium of the embodiment of the present application includes a computer program. When the computer program is executed by a processor, the processor executes the training method of any of the above embodiments.

[0009] The computer program product of the embodiment of the present application includes a computer program, and when the computer program is executed by a processor, the training method described in any one of the above embodiments is implemented.

[0010] In the training method based on extended reality device, the training apparatus based on extended reality device, the extended reality device, the training system, the non-volatile computer-readable storage medium and the computer program product of the implementation mode of the present application, the equipment to be repaired and the digital twin model are superimposed and displayed according to the location information of the equipment to be repaired and the extended reality device, so that the trainee can repair the equipment to be repaired based on the digital twin model. Then, in response to the interactive operation of the digital twin model, the guidance information of the equipment to be repaired is obtained, and then the trainee can learn the maintenance knowledge of the equipment to be repaired based on the guidance steps in the guidance information, and repair the equipment to be repaired to complete the maintenance training. In this way, the present application can complete the training of the trainee based on the extended reality device without taking up more time and energy of the training instructor, thereby reducing the teaching cost of the trainee.

[0011] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0013] Figure 1 is a flow chart of a training method of certain embodiments of the present application;

[0014] Figure 2 is a schematic diagram of the structure of an extended reality device in some embodiments of the present application;

[0015] Figure 3 It is a schematic diagram of a scenario of a training method in certain embodiments of the present application;

[0016] Figure 4 is a flow chart of a training method of certain embodiments of the present application;

[0017] Figure 5 is a flow chart of a training method of certain embodiments of the present application;

[0018] Figure 6 It is a schematic diagram of a scenario of a training method in certain embodiments of the present application;

[0019] Figure 7 is a flow chart of a training method of certain embodiments of the present application;

[0020] Figure 8 It is a schematic diagram of a scenario of a training method in certain embodiments of the present application;

[0021] Fig. 9 It is a schematic diagram of a scenario of a training method in certain embodiments of the present application;

[0022] Fig.10 is a flow chart of a training method of certain embodiments of the present application;

[0023] Fig.11 is a flow chart of a training method of certain embodiments of the present application;

[0024] Fig.12 is a flow chart of a training method of certain embodiments of the present application;

[0025] Fig.13 is a flow chart of a training method of certain embodiments of the present application;

[0026] Fig.14 It is a schematic diagram of a scenario of a training method in certain embodiments of the present application;

[0027] Fig.15 It is a module schematic diagram of a training device based on an extended reality device in certain embodiments of the present application;

[0028] Fig.16 is a schematic diagram of the connection state of a non-volatile computer-readable storage medium and a processor in certain embodiments of the present application;

[0029] Fig.17 It is a schematic diagram of the connection status of a computer program product and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0030] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0031] See also Figures 1 to 3 The present application embodiment provides a training method based on an extended reality device, the training method comprising:

[0032] Step 011: Based on the location information of the device to be repaired 200 and the extended reality device 100, the digital twin model 201 of the device to be repaired 200 and the device to be repaired 200 are superimposed and displayed;

[0033] Among them, the digital twin makes full use of physical models, sensor updates, operation history and other data, integrates multi-disciplinary, multi-physical quantity, multi-scale, and multi-probability simulation processes, and completes mapping in the virtual space, thereby reflecting the entire life cycle process of the corresponding physical equipment to be repaired 200.

[0034] The equipment to be repaired 200 may be HVAC equipment, which includes heating, ventilation, and air conditioning. HVAC equipment is equipment related to heating, ventilation, and air conditioning, such as air conditioners, boilers, radiators, electric heaters, etc. The equipment to be repaired 200 may also be household appliances, such as washing machines, water heaters, refrigerators, etc. Among them, repair is the abbreviation of maintenance and repair. It can be understood that in the operation process of HVAC equipment or household appliances, the maintenance of the equipment is indispensable.

[0035] The digital twin model 201 of the equipment to be repaired 200 is the mapping of the equipment to be repaired 200 in three-dimensional space. The digital twin model 201 can realize simulation operation. The size of the digital twin model 201 and the equipment to be repaired 200 is 1:1. The operating parameters of the digital twin model 201 during simulation operation are consistent with the operating parameters of the equipment to be repaired 200 during actual operation, and the parameters of the sensors in the digital twin model 201 are consistent with the parameters of the sensors corresponding to the equipment to be repaired 200. By checking the working conditions of the digital twin model 201, the actual working conditions of the equipment to be repaired 200 can be obtained.

[0036] Extended Reality (XR) is a general term for multiple technologies such as Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). Extended Reality specifically refers to combining real scenes with virtual scenes through computers to create a virtual environment that allows human-computer interaction. Through extended reality devices 100 (such as XR glasses), users can integrate virtual content with real scenes and have a sense of immersion with seamless transitions between the virtual world and the real world.

[0037] Specifically, during training, the trainee can use the extended reality device 100 and display the digital twin model 201 on the display interface T of the display 20 of the extended reality device 100. Then, through the extended reality device 100, the digital twin model 201 is superimposed on the device to be repaired 200 in the real scene for display, so that the guidance steps in the guidance information subsequently obtained correspond to the device to be repaired 200 in the real scene. For example, if the guidance information is to display an operation gesture on the digital twin model 201, the trainee can imitate the displayed operation gesture at the corresponding position of the device to be repaired 200 to repair the device to be repaired 200, so that the trainee can learn the maintenance knowledge of the device to be repaired 200 based on the digital twin model 201 and repair the device to be repaired 200.

[0038] Step 012: In response to the interactive operation on the digital twin model 201 , the guidance information of the equipment to be repaired 200 is obtained, and the guidance information at least includes one or more guidance steps for repairing the equipment to be repaired 200 .

[0039] Specifically, when the trainee uses the extended reality device 100, the trainee can interact with the digital twin model 201, such as clicking on the digital twin model 201, to indicate that the trainee is ready and can start training. Then the extended reality device 100 will respond to the interactive operation on the digital twin model 201 and obtain guidance information including at least one or more guidance steps for repairing the equipment to be repaired 200. When the guidance information is obtained, the trainee can learn how to repair the equipment to be repaired 200 through the guidance information, and perform actual repair operations on the equipment to be repaired 200 in the real scene, so as to enhance the ability to apply maintenance knowledge and improve the trainee's ability to repair the equipment to be repaired 200.

[0040] Furthermore, in the case where there are multiple guidance steps, all the guidance steps can be acquired at once, so that the trainee can directly see the complete steps of repairing the equipment to be repaired 200. Alternatively, the guidance steps can also be acquired sequentially, and only one guidance step is acquired at a time. After the trainee performs the corresponding maintenance operation on the equipment to be repaired 200 according to the current guidance step, the extended reality device 100 acquires the next guidance step to enhance the trainee's understanding of each guidance step.

[0041] The training method of the embodiment of the present application superimposes and displays the equipment to be repaired 200 and the digital twin model 201 according to the location information of the equipment to be repaired 200 and the extended reality device 100, so that the trainee can repair the equipment to be repaired 200 based on the digital twin model 201. Then, in response to the interactive operation of the digital twin model 201, the guidance information of the equipment to be repaired 200 is obtained, and then the trainee can learn the maintenance knowledge of the equipment to be repaired 200 based on the guidance steps in the guidance information, and repair the equipment to be repaired 200 to complete the maintenance training. In this way, the present application can complete the training of the trainee based on the extended reality device 100 without taking up more time and energy of the training instructor, thereby reducing the teaching cost of the trainee.

[0042] See also Figure 2 , Figure 3 and Figure 4 , training methods may include:

[0043] Step 013: Identify the equipment to be repaired 200 to obtain a digital twin model 201 of the equipment to be repaired 200. The digital twin model 201 is generated based on the three-dimensional model and operation information of the equipment to be repaired 200.

[0044] Specifically, there are multiple methods for obtaining the digital twin model 201 of the equipment to be repaired 200. In one embodiment, the corresponding digital twin model 201 can be pre-established according to the equipment to be repaired 200. When establishing the digital twin model 201, first obtain the three-dimensional image and corresponding operation information of the corresponding equipment to be repaired 200. Then, combine the three-dimensional image and the operation information to make the three-dimensional model simulation run, thereby realizing the establishment of the digital twin model 201.

[0045] Then, during training, the trainee needs to identify the equipment to be repaired 200 through the extended reality device 100. The extended reality device 100 will obtain an image of the equipment to be repaired 200 and determine the model of the equipment to be repaired 200 based on the image of the equipment to be repaired 200, thereby obtaining the corresponding digital twin model 201 to ensure the accuracy of the acquisition of the digital twin model 201.

[0046] In another embodiment, the extended reality device 100 can be connected to the equipment to be repaired 200 for communication, and the digital twin model 201 can be generated in real time by the extended reality device 100. The extended reality device 100 first obtains an image of the equipment to be repaired 200, identifies the model of the equipment to be repaired 200, and obtains the corresponding three-dimensional model according to the model of the equipment to be repaired 200. Then the extended reality device 100 obtains the real-time operation information of the equipment to be repaired 200 through communication with the equipment to be repaired 200. After obtaining the operation information, the digital twin model 201 can be generated in real time according to the operation information and the three-dimensional model, so as to ensure that the digital twin model 201 superimposed on the equipment to be repaired 200 can meet the actual operation of the equipment to be repaired 200.

[0047] See also Figure 2 , Figure 3 and Figure 5 In some embodiments, the device to be repaired 200 includes one or more components to be repaired 202, and the guidance information includes a three-dimensional holographic image of the component to be repaired 202. Step 012: in response to the interactive operation on the digital twin model 201, obtaining the guidance information of the device to be repaired 200 may include:

[0048] Step 0121: In response to an interactive operation on the component to be repaired 202, a three-dimensional holographic image of the component to be repaired 202 is displayed.

[0049] Specifically, the equipment to be repaired 200 includes one or more components to be repaired 202. For example, when the equipment to be repaired 200 is an air conditioner, the components to be repaired 202 may be a motor or a condenser. In order to enable the trainees to cope with most maintenance situations after training, the trainees need to understand the maintenance method of each component of the equipment to be repaired 200 as much as possible during the training process.

[0050] The digital twin model 201 includes one or more parts to be repaired 202 of the corresponding equipment to be repaired 200, and the training object can see one or more parts to be repaired 202 in the digital twin model 201 through the extended reality device 100. Then, during the training, the training object can interact with a part to be repaired 202 in the digital twin model 201, such as clicking on a part to be repaired 202 in the digital twin model 201, and then the extended reality device 100 will respond to the interactive operation on the part to be repaired 202 and display the three-dimensional holographic image of the part to be repaired 202. For example Figure 6When the trainee clicks on the motor of the repaired component 202, the extended reality device 100 may display a three-dimensional holographic image of the motor, a disassembly diagram of the motor, and corresponding operation information. In this way, the trainee can understand the structure of the repaired component 202 in detail based on the three-dimensional holographic image of the repaired component 202, thereby facilitating the trainee to disassemble and repair the repaired component 202.

[0051] See also Figure 2 , Figure 3 , Figure 7 and Figure 8 The guidance information at least includes text guidance information, voice guidance information and image guidance information. Step 012: in response to the interactive operation on the digital twin model 201, the guidance information of the equipment to be repaired 200 is obtained, which may also include:

[0052] Step 0122: In response to interactive operations on the digital twin model 201, text guidance information and image guidance information are displayed, and voice guidance information is issued.

[0053] Specifically, the guidance information includes at least one of text guidance information, voice guidance information and image guidance information.

[0054] For example, in one embodiment, the guidance information is one type, for example, the guidance information is image guidance information, and the image guidance information is a navigation path and a corresponding operation gesture, so as to display a navigation path from the current position of the subject to be trained to the position of the component to be repaired 202 according to the navigation path, and then display the guidance information of the component to be repaired 202 through the operation gesture to guide the subject to be trained to repair the component to be repaired 202. At this time, there are many ways to display the guidance information, for example, the extended reality device 100 can display the guidance information of multiple components to be repaired 202 at one time, for example Figure 8 The guidance information of the repaired component B and the repaired component D is displayed in the extended reality device 100; the guidance steps of the repaired component 202 can also be displayed in sequence. Fig. 9 , the guidance steps of the component to be repaired B are respectively guidance step E and guidance step F. When the subject to be trained selects the component to be repaired B, the extended reality device 100 can enlarge the display of the component to be repaired B, and then display guidance step E. When the subject to be trained completes guidance step E, guidance step F is displayed. Of course, the guidance information can also be voice guidance information. The extended reality device 100 guides the subject to be trained to move to the position of the component to be repaired 202 by voice playback, and plays the maintenance method of the component to be repaired 202 by voice, so as to train the subject to be trained.

[0055] In another embodiment, the guidance information may be two of text guidance information, voice guidance information, and image guidance information. For example, the guidance information is image guidance information and text guidance information, wherein the image guidance information includes a navigation path, and the training object can be guided to move to the position of the component to be repaired 202 through the navigation path, and the maintenance method of the component to be repaired 202 is displayed in the form of text. For another example, the guidance information may include voice guidance information and image guidance information at the same time, wherein the image guidance information includes a navigation path and a corresponding operation gesture, and the training object can be guided to move to the position of the component to be repaired 202 through the navigation path, and the maintenance method of the component to be repaired 202 is displayed by displaying the corresponding operation gesture, and the maintenance method of the component to be repaired 202 is played by voice at the same time.

[0056] Of course, in another embodiment, the guidance information may include text guidance information, voice guidance information and image guidance information. For example, the image guidance information is used to guide the trainee to move to the position of the component to be repaired 202, and then the maintenance method of the component to be repaired 202 is displayed in text form, and the maintenance method of the component to be repaired 202 is played by voice at the same time, so as to further enhance the trainee's understanding of the maintenance method.

[0057] In this way, the extended reality device 100 can use various forms of guidance information to guide the trainees, so as to provide guidance to the trainees from different angles, and try to ensure that the trainees can understand the guidance information, thereby improving the training effect of the trainees.

[0058] See also Figure 3 and Fig.10 In some embodiments, the training method may further include:

[0059] Step 014: Determine the maintenance score of each component 202 to be maintained according to the completion time of the guidance steps of each component 202 to be maintained and the maintenance completion degree;

[0060] Step 015: Determine the target component 202 to be repaired according to the repair score of each component 202 to be repaired;

[0061] Step 012: In response to the interactive operation on the digital twin model 201, obtaining the guidance information of the equipment to be repaired 200 may also include:

[0062] Step 0123: In response to the interactive operation on the target component to be repaired 202, the guidance information of the target component to be repaired 202 is obtained.

[0063] Specifically, in order to check the maintenance learning effect of the trainee for each repaired component 202, after the trainee repairs the repaired component 202 according to the guidance steps, the maintenance effect of the trainee can also be scored. The judgment criteria of the maintenance score may include the completion time of the guidance steps and the maintenance completion degree.

[0064] Please combine Figure 1 When the trainee is undergoing training, the extended reality device 100 can record the completion time of each guide step and determine the maintenance completion degree of each guide step. Then, the maintenance score of each component to be repaired 202 is determined according to the completion time of the guide step and the maintenance completion degree of each component to be repaired 202. After obtaining the maintenance score of each component to be repaired 202, the target component to be repaired 202 that needs to be focused on can be determined therefrom, for example, the components to be repaired 202 with the lowest scores can be used as the target components to be repaired 202. After determining the target component to be repaired 202, the target component to be repaired 202 can be highlighted in the display page, for example, the target component to be repaired 202 can be marked in red to remind the trainee that the target component to be repaired 202 needs to be focused on, or the number of training times for the target component to be repaired 202 can be directly increased to ensure that the trainee's maintenance ability for the target component to be repaired 202 can be fully exercised. Then, when the subject to be trained performs an interactive operation on the target component to be repaired 202 , the extended reality device 100 obtains guidance information of the target component to be repaired 202 in response to the interactive operation on the target component to be repaired 202 to guide the subject to be trained to learn and complete the repair of the target component to be repaired 202 .

[0065] Furthermore, in one embodiment, when determining the maintenance score of each component to be repaired 202, the maintenance score of each component to be repaired 202 of each training object can be determined. At this time, the target component to be repaired 202 of each training object can be determined in a targeted manner according to the maintenance score of each component to be repaired 202 of each training object, so as to achieve personalized targeted training, thereby training the training object's maintenance ability for the target component to be repaired 202 with poor personal maintenance effect as much as possible. Figure 3 For example, if the repair score of the training subject A for the repair component B is low, and the repair score of the training subject C for the repair component D is low, then the repair component B can be set as the target repair component 202 for the training subject A, and the repair component D can be set as the target repair component 202 for the training subject C.

[0066] In another embodiment, the maintenance score of the component to be repaired 202 can be determined based on the maintenance score of each subject to be trained on the same component to be repaired 202. For example, the average score of the maintenance score of each subject to be trained on the component to be repaired B is used as the maintenance score of the component to be repaired B. If the maintenance score of a component to be repaired 202 is low, it means that the maintenance effect of the component to be repaired 202 is generally poor and the maintenance difficulty is high. At this time, each subject to be trained can be required to perform intensive training on the component to be repaired 202. Therefore, based on the maintenance score of each component to be repaired 202, the component to be repaired 202 with generally poor maintenance effect is obtained, and the component to be repaired 202 with generally poor maintenance effect is confirmed as the target component to be repaired 202, so that each subject to be trained can perform targeted intensive training on the target component to be repaired 202.

[0067] At present, engineers with strong professional ability are generally used as training instructors to train trainees, and usually the number of trainees is greater than the number of training instructors. Due to the limited energy of the training instructor, it is difficult for the training instructor to fully control the maintenance situation of each trainee for each guide step, making it difficult to monitor the training effect of each trainee in real time, and thus it is difficult to optimize the training plan in time according to the training effect of each trainee. However, the present application can determine the training effect of the trainee for each component to be repaired 202 according to the completion time and maintenance completion degree of the guide step of each component to be repaired 202, and determine the maintenance score of each component to be repaired 202, so as to set the target component to be repaired 202 in a targeted manner, thereby realizing real-time optimization of the training plan, ensuring the adaptability of the training plan to the trainee, and ensuring as much as possible that after training based on the optimized training plan, the trainee's maintenance ability for each component to be repaired 202 is improved.

[0068] See also Figure 3 and Fig.11 In some embodiments, the guidance information of each component to be repaired 202 includes a preset number of training times, and the preset number of training times is configured as the number of times the component to be repaired 202 is repaired when the component to be repaired 202 completes the training; the training method also includes:

[0069] Step 016: adjusting the preset number of training times according to the maintenance scores of the components to be repaired 202 corresponding to the multiple training subjects.

[0070] Specifically, during training, for a component to be repaired 202, the trainee can undergo multiple maintenance trainings. When the number of times the component to be repaired 202 has been repaired reaches a preset number of trainings, it is determined that the training for the component to be repaired 202 has been completed. Then, after obtaining the maintenance scores of each component to be repaired 202 corresponding to the scores of multiple trainees, the preset number of trainings for each component to be repaired 202 can be adjusted according to the maintenance scores.

[0071] For example, for a component 202 to be repaired with a low maintenance score and generally poor maintenance effect, the preset number of training times for the component 202 to be repaired can be increased to adaptively increase the amount of training for the component 202 to be repaired with poor maintenance effect. Alternatively, the preset number of training times for a component 202 to be repaired with a high maintenance score can be reduced to avoid the trainee spending too much time on the component 202 to be repaired with a high maintenance score and a more mature maintenance capability, thereby increasing unnecessary training costs for the trainee.

[0072] When the preset training times for the repaired component 202 with a lower repair score are increased, the repair score of the repaired component 202 is improved after enhanced training. At this time, the preset training times for the repaired component 202 can also be reduced, and the preset training times can be adjusted in time according to the repair score of the repaired component 202.

[0073] In addition, a minimum number of training times for the preset training times can also be set. When reducing the preset number of training times for a certain repaired component 202, the preset number of training times cannot be lower than the minimum number of training times. Even if the score of the repaired component 202 is high and the corresponding preset number of training times is continuously reduced, the preset number of training times for the repaired component 202 can only be the minimum number of training times at least, so that the trainee still needs to be trained according to the minimum number of training times during the training, so as to consolidate the repair ability of the trainee for the repaired component 202 and prevent the repair ability from declining when the preset number of training times is too small.

[0074] In this way, the preset number of training times for each component to be repaired 202 can be adjusted in a targeted manner according to the maintenance scores of each component to be repaired 202 corresponding to multiple trainees, so as to ensure that the trainees' maintenance capabilities for each component to be repaired 202 can be fully exercised on the one hand, and on the other hand, minimize unnecessary time expenditure and training cost expenditure for the trainees.

[0075] See also Figure 3 and Fig.12 In some embodiments, the maintenance score includes a first score and a second score. Step 014: determining the maintenance score of each component to be repaired 202 according to the completion time of the guide step of each component to be repaired 202 and the maintenance completion degree may include:

[0076] Step 0141: determining a first score according to the total completion time of each guide step of each component to be repaired 202, wherein the first score is negatively correlated with the total completion time;

[0077] Step 0142: Determine the maintenance completion degree of each guiding step according to the image and operation information of the component to be repaired 202 after each guiding step is completed;

[0078] Step 0143: determining the repair completion degree of the component to be repaired 202 according to the repair completion degree of each guided step, and determining a second score according to the repair completion degree of the component to be repaired 202, wherein the second score is positively correlated with the repair completion degree of the component to be repaired 202;

[0079] Step 0144: Determine the maintenance score of each component 202 to be repaired according to the first score and the second score of each component 202 to be repaired.

[0080] Specifically, during the training, the completion time of each guide step of each component to be repaired 202 and the image and operation information of the component to be repaired 202 after each guide step are completed can be counted. Then, the first score is determined according to the total completion time of each guide step of each component to be repaired 202. It can be understood that the longer the total completion time, the less proficient the trainee is in repairing the component to be repaired 202, and the lower the first score. Then, according to the image and operation information of the component to be repaired 202 after each guide step is completed, the repair completion degree of each guide step is determined. For example, after the trainee repairs the component to be repaired 202, the image and operation information of the component to be repaired 202 show that the repair of the trainee has achieved the preset effect. At this time, it can be confirmed that the repair completion degree of the guide step is 100%. After determining the maintenance completion degree of each component to be repaired 202, the maintenance completion degree of the component to be repaired 202 can be determined, and the second score can be determined according to the maintenance completion degree of the component to be repaired 202. It can be understood that the higher the maintenance completion degree, the more proficient the trainee is in repairing the component to be repaired 202, and the higher the second score. After obtaining the first score and the second score of each component to be repaired 202, the maintenance score of each component to be repaired 202 can be determined, for example, after simply adding the first score and the second score, the added value is used as the maintenance score, or the average of the first score and the second score is taken as the maintenance score.

[0081] In particular, when determining the maintenance completion degree of the component to be repaired 202 based on the maintenance completion degree of each guiding step, the arithmetic mean of the maintenance completion degrees of each guiding step can be directly taken as the maintenance completion degree of the component to be repaired 202, or the maintenance completion degree of each guiding step can be weighted averaged, that is, according to the importance of each guiding step, a corresponding weight is set, and then the maintenance completion degree of the component to be repaired 202 is calculated based on the maintenance completion degree of each guiding step and the corresponding weight, so as to take the importance of different guiding steps into account by weight, thereby obtaining a more accurate and reliable maintenance completion degree. For example, the guidance steps for the component to be repaired 202 are divided into three steps. The maintenance completion degree of the first step is 100%, and the corresponding weight is 0.25. The maintenance completion degree of the second step is 90%, and the corresponding weight is 0.35. The maintenance completion degree of the first step is 80%, and the corresponding weight is 0.4. Then the calculation formula for the maintenance completion degree of the component to be repaired 202 is 100%*0.25+90%*0.35+80%*0.4, and the final maintenance completion degree of the component to be repaired 202 is 88.5%.

[0082] Similarly, when calculating the total completion time of each guiding step of each component to be repaired 202 based on the completion time of the guiding step of each component to be repaired 202, the completion time of the guiding step of each component to be repaired 202 can be directly added to obtain the total completion time; or a corresponding weight can be set for the guiding step of each component to be repaired 202, and a weighted sum can be performed to obtain the total completion time after weighted calculation, so as to take the importance of different guiding steps into account by means of weights, thereby obtaining a more accurate and reliable total completion time.

[0083] In this way, the accurate maintenance score of each component to be repaired 202 can be determined based on the completion time of each guidance step of each component to be repaired 202 and the image and operation information of the component to be repaired 202 after each guidance step is completed, so as to subsequently determine the training effect of each component to be repaired 202 according to the accurate maintenance score and optimize the training plan based on the training effect of each component to be repaired 202.

[0084] See also Figure 2 and Fig.13 In some embodiments, the training method further comprises:

[0085] Step 017: Displaying an image within the field of view of the extended reality device 100 in an electronic device other than the extended reality device 100.

[0086] Specifically, please combine Figure 3During the training process, a training instructor can also be configured to assist the extended reality device 100 in training the trainee. In order to facilitate the training instructor to obtain the training status of the trainee, the perspective of the trainee can be shared at this time, that is, the image within the field of view of the extended reality device 100 is displayed in an electronic device other than the extended reality device 100, such as XR glasses, mobile phones, computers, smart watches, etc., which have a display 20, for example Fig.14 , to synchronously display content (such as on-site scenes, digital twin models 201 and equipment to be repaired 200) and interactive operations to the electronic device of the training instructor, so that the training instructor can monitor the training status of the trainee according to the images displayed in the electronic device, and when necessary, for example, when the trainee is unable to correctly understand the guidance steps, the training instructor can manually teach the trainee according to the training status of the trainee, so as to further provide targeted training to the trainee, thereby improving the training effect. At this time, the training instructor can also judge the training status of the trainee according to the image within the field of view of the extended reality device 100, and score each guidance step of each trainee in a targeted manner, thereby ensuring the accuracy of the maintenance score.

[0087] It can be understood that compared with the current method in which training instructors undertake most of the training work during the training process, the workload of the training instructors in this application is relatively small, and most of the training work can be completed by the extended reality device 100. The training instructors only need to assist the extended reality device 100 in training, thereby avoiding the situation in which more of the training instructors' time and energy are taken up.

[0088] See also Fig.15 In order to better implement the above training method, the embodiment of the present application also provides a training device 10 based on an extended reality device 100. The training device 10 based on the extended reality device 100 may include an overlay module 11 and an acquisition module 12. The overlay module 11 is used to overlay and display the digital twin model 201 of the device to be repaired 200 and the device to be repaired 200 according to the location information of the device to be repaired 200 and the extended reality device 100; the acquisition module 12 is used to respond to the interactive operation of the digital twin model 201, and obtain the guidance information of the device to be repaired 200, and the guidance information at least includes one or more guidance steps for repairing the device to be repaired 200.

[0089] The training device 10 may further include an identification module 13. The identification module 13 is used to identify the equipment to be repaired 200 to obtain a digital twin model 201 of the equipment to be repaired 200. The digital twin model 201 is generated according to the three-dimensional model and operation information of the equipment to be repaired 200.

[0090] The acquisition module 12 is specifically configured to display a three-dimensional holographic image of the component to be repaired 202 in response to an interactive operation on the component to be repaired 202 .

[0091] The acquisition module 12 is specifically used to respond to interactive operations on the digital twin model 201, display text guidance information and image guidance information, and issue voice guidance information.

[0092] The training device 10 further includes a first determination module 14 and a second determination module 15. The first determination module 14 is used to determine the maintenance score of each component to be repaired 202 according to the completion time of the guidance steps and the maintenance completion degree of each component to be repaired 202. The second determination module 15 is used to determine the target component to be repaired 202 according to the maintenance scores of each component to be repaired 202.

[0093] The acquisition module 12 is further specifically configured to acquire the guidance information of the target component to be repaired 202 in response to the interactive operation on the target component to be repaired 202 .

[0094] The training device 10 further includes an adjustment module 16. The adjustment module 16 is used to adjust the preset number of training times according to the maintenance scores of the components to be maintained 202 corresponding to the multiple training subjects.

[0095] The first determination module 14 is also specifically used to determine a first score according to the total completion time of each guidance step of each component to be repaired 202, and the first score is negatively correlated with the total completion time; determine the repair completion degree of each guidance step according to the image and operation information of the component to be repaired 202 after each guidance step is completed; determine the repair completion degree of the component to be repaired 202 according to the repair completion degree of each guidance step, and determine a second score according to the repair completion degree of the component to be repaired 202, and the second score is positively correlated with the repair completion degree of the component to be repaired 202; determine the repair score of each component to be repaired 202 according to the first score and the second score of each component to be repaired 202.

[0096] The training device 10 further includes a display module 17. The display module 17 is used to display an image within the field of view of the extended reality device 100 in an electronic device other than the extended reality device 100.

[0097] Please refer again Figure 2 An embodiment of the present application provides an extended reality device 100, which includes a display 20, one or more processors 30, a memory 40, and one or more computer programs, wherein the one or more computer programs are stored in the memory 40 and executed by the one or more processors 30, and the computer program includes instructions for the training method of any of the above embodiments.

[0098] See also Figure 2 and Figure 3 An embodiment of the present application provides a training system, which includes a device to be repaired 200 and an extended reality device 100. The extended reality device 100 is communicatively connected to the device to be repaired 200, and the extended reality device 100 is used for the steps of the training method of any of the above embodiments.

[0099] See also Figure 2 and Fig.16 The embodiment of the present application also provides a non-volatile computer-readable storage medium 300, on which a computer program 310 is stored. When the computer program 310 is executed by a processor 320, the steps of the training method of any of the above-mentioned embodiments are implemented. The processor 320 can be the processor 30 of the extended reality device 100. For the sake of brevity, it will not be repeated here.

[0100] See also Figure 2 and Fig.17 The embodiment of the present application also provides a computer program product 400, which includes a computer program 410. When the computer program 410 is executed by a processor 420, the training method of any of the above embodiments is implemented, wherein the processor 420 can be the processor 30 in the extended reality device 100 in the above embodiment. For the sake of brevity, it will not be repeated here.

[0101] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0103] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A training method based on an extended reality device, characterized in that: include: According to the location information of the device to be repaired and the extended reality device, a digital twin model of the device to be repaired and the device to be repaired are superimposed and displayed; In response to interactive operations on the digital twin model, guidance information of the device to be repaired is obtained, and the guidance information at least includes one or more guidance steps for repairing the device to be repaired.

2. The training method according to claim 1, characterized in that: Also includes: The equipment to be repaired is identified to obtain the digital twin model of the equipment to be repaired, wherein the digital twin model is generated according to the three-dimensional model and operation information of the equipment to be repaired.

3. The training method according to claim 1, characterized in that: The device to be repaired includes one or more components to be repaired, the guidance information includes a three-dimensional holographic image of the components to be repaired, and the obtaining of the guidance information of the device to be repaired in response to an interactive operation on the device to be repaired includes: In response to an interactive operation on the component to be repaired, the three-dimensional holographic image of the component to be repaired is displayed.

4. The training method according to claim 1 or 3, characterized in that: The guidance information includes at least text guidance information, voice guidance information and image guidance information, and the step of obtaining the guidance information of the equipment to be repaired in response to the interactive operation on the digital twin model includes: In response to an interactive operation on the digital twin model, the text guidance information and the image guidance information are displayed, and the voice guidance information is issued.

5. The training method according to claim 1, characterized in that: The equipment to be repaired includes one or more components to be repaired, and the method further includes: Determine a maintenance score for each component to be repaired according to the completion time of the guidance steps and the degree of maintenance completion of each component to be repaired; Determining a target component to be repaired according to the repair score of each component to be repaired; The step of obtaining guidance information of the device to be repaired in response to an interactive operation on the device to be repaired includes: In response to the interactive operation on the target component to be repaired, guidance information of the target component to be repaired is acquired.

6. The training method according to claim 5, characterized in that: The guidance information of each of the components to be repaired includes a preset number of training times, where the preset number of training times is configured as the number of times the component to be repaired is repaired when the component to be repaired completes the training; The method further comprises: The preset number of training times is adjusted according to the maintenance scores of the components to be repaired corresponding to the multiple training subjects.

7. The training method according to claim 5, characterized in that: The maintenance score includes a first score and a second score, and the maintenance score of each component to be repaired is determined according to the completion time of the guidance step and the maintenance completion degree of each component to be repaired, including: Determine the first score according to the total completion time of each of the guiding steps of each of the components to be repaired, wherein the first score is negatively correlated with the total completion time; Determining the degree of completion of maintenance of each of the guiding steps according to the image and operation information of the component to be repaired after each of the guiding steps is completed; Determining the repair completion degree of the component to be repaired according to the repair completion degree of each of the guiding steps, and determining the second score according to the repair completion degree of the component to be repaired, wherein the second score is positively correlated with the repair completion degree of the component to be repaired; A maintenance score of each of the components to be repaired is determined according to the first score and the second score of each of the components to be repaired.

8. The training method according to claim 1, characterized in that: Also includes: An image within the field of view of the extended reality device is displayed in an electronic device other than the extended reality device.

9. A training device based on an extended reality device, characterized in that: include: A superposition module, used for superimposing and displaying the digital twin model of the device to be repaired and the device to be repaired according to the location information of the device to be repaired and the extended reality device; and An acquisition module is used to acquire guidance information of the device to be repaired in response to an interactive operation on the digital twin model, wherein the guidance information at least includes one or more guidance steps for repairing the device to be repaired.

10. An extended reality device, characterized in that: include: monitor; One or more processors, memory; and One or more computer programs, wherein one or more of the computer programs are stored in the memory and executed by one or more of the processors, and the computer programs include instructions for executing the training method according to any one of claims 1 to 8.

11. A training system, characterized in that: include: Equipment to be repaired; and An extended reality device, wherein the extended reality device is communicatively connected to the equipment to be repaired, and the extended reality device is used to execute the training method according to any one of claims 1 to 8.

12. A non-volatile computer-readable storage medium containing a computer program, wherein when the computer program is executed by a processor, the processor executes the training method according to any one of claims 1 to 8.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the training method according to any one of claims 1 to 8 is implemented.