Steel mill maintenance bench worker training method, device and system based on mixed reality technology

Abstract

The invention provides a mixed reality technology-based steel mill maintenance bench worker training method, device and system, and the method comprises the following steps: obtaining a virtual three-dimensional model which comprises courseware information which at least comprises bench worker operation process information and three-dimensional operation action information; obtaining a physical three-dimensional model on the interactive equipment, wherein the physical three-dimensional model specifically comprises an equipment model of bench worker operation entity equipment; virtual-real registration is carried out on the physical three-dimensional model based on the virtual three-dimensional model, and the virtual-real registration comprises coordinate matching and model superposition; feedback signals on interaction equipment are received for digital interaction so as to complete bench worker training, and the interaction process comprises the step of conducting training control on the virtual three-dimensional model. According to the steel mill maintenance bench worker training method, device and system based on the mixed reality technology, immersive virtual training of a steel mill equipment maintenance process is realized, and the intelligence level of training teaching is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of virtual reality technology, and particularly relates to a steel plant maintenance bench worker training method, device and system based on a mixed reality technology. BACKGROUND

[0002] Steel plant equipment has the characteristics of complex equipment failure and strong professionalism in operation and maintenance. In actual application, the technical inheritance problem caused by personnel retirement is increasingly prominent, and new employees also need pre-service skill training, so as to ensure the normal operation of the equipment.

[0003] Among them, the existing human training method has certain shortcomings, and the specific defects are as follows: (1) limited teacher resources and high cost of teaching aids; (2) complex knowledge explanation is not vivid and image, and there are more theories and less practical operations; (3) lack of intuitive experience in teaching, and the training effect of students is not ideal. SUMMARY

[0004] In view of the above-mentioned defects of the prior art, the purpose of the present application is to provide a steel plant maintenance bench worker training method, device and system based on a mixed reality technology, which is used to solve the problem of steel plant maintenance bench worker training based on a mixed reality technology.

[0005] In a first aspect, the present application provides a steel plant maintenance bench worker training method based on a mixed reality technology, which comprises the following steps:

[0006] Obtaining a virtual three-dimensional model, the virtual three-dimensional model comprising courseware information, the courseware information at least comprising bench worker operation flow information and three-dimensional operation action information;

[0007] Obtaining a physical three-dimensional model on an interactive device, wherein the physical three-dimensional model specifically comprises a device model of a bench worker operation entity device;

[0008] Performing virtual-real registration on the physical three-dimensional model based on the virtual three-dimensional model, wherein the virtual-real registration comprises coordinate matching and model coincidence;

[0009] Receiving a feedback signal on the interactive device to complete digital interaction for bench worker training, wherein the interaction process comprises training control on the virtual three-dimensional model.

[0010] In a possible implementation manner of the present application, the virtual three-dimensional model is obtained, specifically comprising:

[0011] Obtaining editing data in a preset editing tool, wherein the preset editing tool comprises a three-dimensional courseware editing tool;

[0012] Obtaining an operation flow expression based on the editing data, wherein the expression content at least includes operation nodes, detection nodes and maintenance nodes;

[0013] Obtaining node operation data based on the editing data, wherein the node operation data at least includes operation content, operation objects, tools used and matters needing attention;

[0014] Obtaining the bench work operation flow information based on the content of the operation flow expression, and obtaining three-dimensional operation action information based on the node operation data.

[0015] In a possible implementation of the present application, the physical three-dimensional model on the interactive device is specifically obtained by:

[0016] Obtaining an initial physical model on the interactive device;

[0017] Performing format export on the initial physical model to obtain a step format feature model;

[0018] Performing lightweight processing on the step format feature model by using a three-dimensional model conversion tool to obtain the physical three-dimensional model, wherein the physical three-dimensional model is an ive format mesh model.

[0019] In a possible implementation of the present application, the method further includes arranging and encrypting the courseware information to form a binary format courseware data file, and storing the courseware data file to a preset position of the interactive device.

[0020] In a possible implementation of the present application, the virtual-real registration based on the virtual three-dimensional model on the physical three-dimensional model specifically includes:

[0021] Performing coordinate matching between the virtual three-dimensional model and the physical three-dimensional model based on a preset adjustment expression until the geometric position error between the virtual three-dimensional model and the physical three-dimensional model falls within a preset error range;

[0022] According to the current control six-degree-of-freedom pose, calculating the relative pose matrix of the registered virtual three-dimensional model and the physical three-dimensional model, and performing model rework based on the relative pose matrix to complete model virtual-real fusion.

[0023] In a possible implementation of the present application, the receiving of the feedback signal on the interactive device for digital interaction to complete bench work training specifically includes:

[0024] Obtaining the feedback signal based on a collection device arranged on the interactive device, wherein the collection device at least includes an image collection device and a microphone;

[0025] Carrying out digital interaction on the virtual three-dimensional model based on the feedback signal, wherein the feedback signal at least includes gesture image signal and voice operation instruction signal.

[0026] In a second aspect, the present application provides a steel plant maintenance benchwork training device based on mixed reality technology, which comprises:

[0027] A course module is configured to acquire a virtual three-dimensional model, wherein the virtual three-dimensional model comprises course information, and the course information at least includes benchwork operation flow information and three-dimensional operation action information.

[0028] An equipment module is configured to acquire a physical three-dimensional model on an interactive device, wherein the physical three-dimensional model specifically comprises a device model of a benchwork operation entity device.

[0029] A registration module is configured to carry out virtual-physical registration on the physical three-dimensional model based on the virtual three-dimensional model, wherein the virtual-physical registration comprises coordinate matching and model coincidence.

[0030] An interaction module is configured to receive a feedback signal on the interactive device to carry out digital interaction and complete benchwork training, wherein the interaction process comprises training control on the virtual three-dimensional model.

[0031] In a third aspect, the present application provides an electronic device, which comprises a processor and a memory.

[0032] The memory is configured to store a computer program.

[0033] The processor is configured to execute the computer program stored in the memory, so that the electronic device executes the above-mentioned steel plant maintenance benchwork training method based on mixed reality technology.

[0034] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by an electronic device to implement the above-mentioned steel plant maintenance benchwork training method based on mixed reality technology.

[0035] In a fifth aspect, the present application provides a steel plant maintenance benchwork training system based on mixed reality technology, which comprises:

[0036] An entity device comprises a benchwork operation device, which is configured to provide a physical three-dimensional model obtained by three-dimensional modeling by a user.

[0037] An interactive device comprises an MR device, which is configured to carry out digital interaction on the physical three-dimensional model by user interaction data as a medium.

[0038] The above-mentioned electronic device is configured to complete the interaction control of benchwork training.

[0039] As described above, the steel plant maintenance fitter training method, device and system based on the mixed reality technology have the following beneficial effects:

[0040] (1) The immersion type virtual training of the steel plant equipment maintenance process can be realized, thereby improving the intelligent level of the steel plant equipment maintenance training teaching;

[0041] (2) The training time of the new employee (fitter) related maintenance skills is effectively shortened, and the training cost investment of the enterprise to the new employee (fitter) is reduced;

[0042] (3) The new employee (fitter) can quickly master the process technology and safety and quality points of the post operation, so as to ensure that the operating personnel strictly operate according to the standardized specification in the production process. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A scene schematic diagram in an embodiment of the electronic device of the present application is shown;

[0044] Figure 2 A method step schematic diagram in an embodiment of the steel plant maintenance fitter training method based on the mixed reality technology of the present application is shown;

[0045] Figure 3 A method step schematic diagram in an embodiment of the steel plant maintenance fitter training method based on the mixed reality technology of the present application is shown;

[0046] Figure 4 A visualization schematic diagram of the three-dimensional courseware editing tool in an embodiment of the steel plant maintenance fitter training method based on the mixed reality technology of the present application is shown;

[0047] Figure 5 A display interface schematic diagram of the intelligent fitter training teaching courseware in an embodiment of the steel plant maintenance fitter training method based on the mixed reality technology of the present application is shown;

[0048] Figure 6 A method step schematic diagram in an embodiment of the steel plant maintenance fitter training method based on the mixed reality technology of the present application is shown;

[0049] Figure 7 A structure schematic diagram in an embodiment of the steel plant maintenance fitter training device based on the mixed reality technology of the present application is shown;

[0050] Figure 8 A structure schematic diagram in an embodiment of the electronic device of the present application is shown;

[0051] Figure 9 A structure schematic diagram in an embodiment of the steel plant maintenance fitter training system based on the mixed reality technology of the present application is shown.

[0052] Element number explanation

[0053] S202-S208 steps

[0054] S302-S308 steps

[0055] S602-S606 steps

[0056] 70 Steel plant maintenance fitter training system based on mixed reality technology

[0057] 71 Courseware module

[0058] 72 Equipment module

[0059] 73 Registration module

[0060] 74 Interaction module

[0061] 90 Steel plant maintenance fitter training system based on mixed reality technology 90

[0062] 91 Physical device

[0063] 92 Interaction device

[0064] 93 Electronic device DETAILED DESCRIPTION

[0065] The present application will be described in detail below with specific embodiments, and other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the present application. The present application can also be implemented or applied by other different embodiments, and various modifications or changes can be made to the details in the present specification based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0066] It should be noted that the drawings provided in the following embodiments only schematically illustrate the basic concept of the present application, and only the components related to the present application are shown in the drawings, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change in shape, number and proportion, and the layout pattern of the components may also be more complex.

[0067] The following embodiments of the present application provide a steel plant maintenance fitter training method based on mixed reality technology, which can be applied to, for example, Figure 1The electronic device shown in the figure. The electronic device described in the present application can include a mobile phone 11, a tablet computer 12, a notebook computer 13, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. with wireless charging function, and the specific type of the electronic device is not limited in the embodiments of the present application.

[0068] For example, the electronic device can be a station (STATION, ST) in a WLAN with wireless charging function, a cellular phone with wireless charging function, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA) device, a handheld device with wireless charging function, a computing device or other processing device, a computer, a laptop, a handheld communication device, a handheld computing device, and / or other equipment for communicating over a wireless system and next-generation communication systems, such as a mobile terminal in a 5G network, a mobile terminal in a future evolved public land mobile network (PLMN), or a mobile terminal in a future evolved non-terrestrial network (NTN), etc.

[0069] For example, the electronic device can communicate with a network and other devices by wireless communication. The above-described wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), Bluetooth (BT), GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou navigation satellite system (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0070] The traditional method relies too much on the experience teaching of the workers, and the standardization and operation standard are difficult to unify, it is difficult to grasp the quality of teaching and training, and the abilities of different new workers (fitter) brought up by different training personnel are uneven. Therefore, the training of the just overhauled fitter needs to introduce a more scientific and comprehensive auxiliary method to convert the valuable experience of the enterprise (steel plant) into digital results and provide immersive learning and training experience for new workers (fitter), so as to improve the skill level of new workers (fitter) and improve the training effect. Among them, the application relates to the application of MR technology in the operation and maintenance of steel plant equipment, and provides a training method for the operation and maintenance of steel plant equipment based on MR (Mix Reality) mixed reality technology. The mixed reality technology (MR) is a further development of virtual reality technology. The technology introduces real scene information into the virtual environment, builds an interactive feedback information loop between the virtual world, the real world and the user, enhances the realism of user experience, and realizes immersive virtual training of the steel plant equipment maintenance process, improves the intelligent level of training and teaching, effectively shortens the training time of new workers (fitter), reduces the training cost investment of the enterprise (steel plant) for new workers (fitter), enables new workers (fitter) to quickly master the post operation process technology and safety and quality points, and strictly operates according to the standardized specification in the future operation.

[0071] The core of the method of the application is to use MR mixed reality technology. During the training of new workers (fitter), the steel plant equipment model in the actual operation scene is introduced into the virtual environment to use digital information for training, which can enhance the realism of user experience. In the teaching process, new workers (fitter) can operate at the same time, carry out digital interaction, enhance communication with each other, find problems in time, and solve problems in the training process, so as to find and solve problems in time. The MR mixed reality technology builds a digital interactive feedback mechanism information loop between virtual-reality-user, which can improve the training efficiency and the accumulation of new workers (fitter) in the operation and maintenance.

[0072] The technical solutions in the embodiments of the application will be described in detail below with reference to the drawings in the embodiments of the application.

[0073] Specifically, refer to Figure 2 In an embodiment of the application, the steel plant maintenance fitter training method based on the mixed reality technology of the application comprises the following steps:

[0074] Step S202, a virtual three-dimensional model is obtained, the virtual three-dimensional model comprises courseware information, and the courseware information at least comprises fitter operation flow information and three-dimensional operation action information;

[0075] In step S204, a physical three-dimensional model on the interactive device is acquired, wherein the physical three-dimensional model specifically comprises a device model of the bench worker operation entity device;

[0076] In step S206, virtual-real registration is performed on the physical three-dimensional model based on the virtual three-dimensional model, wherein the virtual-real registration comprises coordinate matching and model coincidence;

[0077] In step S208, a feedback signal on the interactive device is received to perform digital interaction to complete the bench worker training, wherein the interaction process comprises training control on the virtual three-dimensional model.

[0078] It should be noted that in the foregoing description, examples are given in the present embodiment. The present application uses MR mixed reality technology to set up an information loop of a digital interaction feedback mechanism between virtual-real-users, and thus the virtual three-dimensional model in the virtual environment and the physical three-dimensional model in the real environment need to be acquired, and digital interaction is performed through the feedback signal of the user to complete the entire training process of the new employee (bench worker). After the virtual three-dimensional model and the physical three-dimensional model are acquired, virtual-real registration needs to be performed, so as to guarantee the accuracy and timeliness of the digital interaction of the new employee (bench worker) in the training process.

[0079] Specifically, in one embodiment of the present application, the virtual three-dimensional model is acquired as follows: Figure 3 As shown in the figure, the method specifically comprises the following steps:

[0080] In step S302, editing data in a preset editing tool is acquired, wherein the preset editing tool comprises a three-dimensional courseware editing tool;

[0081] In step S304, an operation flow expression is obtained based on the editing data, wherein the expression content at least comprises an operation node, a detection node and a maintenance node;

[0082] In step S306, node operation data is obtained based on the editing data, wherein the node operation data at least comprises operation content, operation object, tool used and matters needing attention;

[0083] In step S308, the bench worker operation flow information is obtained based on the content of the operation flow expression, and three-dimensional operation action information is obtained based on the node operation data.

[0084] Specifically, in the present embodiment, when applied, the user edits in the preset editing tool to finally obtain corresponding courseware information. Specifically, first, editing data in the preset editing tool is acquired as follows: Figure 4As shown, a visual schematic diagram of a three-dimensional courseware editing tool is shown, specifically an interactive electronic manual editor for user editing, and thus, after obtaining the editing data, operation flow expression and node operation data are obtained based on the editing data, wherein the expression content at least includes operation nodes, detection nodes and maintenance nodes, and the node operation data at least includes operation content, operation objects, tools used and matters needing attention. Each type of data corresponds to the content in the courseware information. Specifically, the bench work operation flow information is obtained based on the content of the operation flow expression, and the three-dimensional operation action information is obtained based on the node operation data. Thus, after obtaining the bench work operation flow information and the three-dimensional operation action information, the courseware information can be obtained. Correspondingly, the courseware information is the basic element for constructing the virtual three-dimensional model.

[0085] Further, in an embodiment of the application, the method further comprises arranging and encrypting the courseware information to form a binary format courseware data file, and storing the courseware data file in a preset position of the interactive device.

[0086] It should be noted that, in this embodiment, after obtaining the courseware information based on the editing data selected or input by the user, the completed courseware information can be packaged and arranged, and encrypted to obtain a courseware data file, wherein the courseware data file is a binary file, which is convenient for transmission and storage. Thus, after obtaining the encrypted courseware data file, it can be stored in the interactive device, such as a storage unit arranged on the interactive device, so that when a new employee (bench worker) uses the interactive device, the corresponding courseware information of the virtual three-dimensional model can be applied in time, wherein, for example, Figure 5 As shown, a display interface schematic diagram of an intelligent bench worker training teaching courseware is shown.

[0087] Further, in an embodiment of the application, as shown in Figure 6 The physical three-dimensional model on the interactive device is obtained, specifically including the following steps:

[0088] Step S602, obtaining an initial physical model on the interactive device;

[0089] Step S604, analyzing the initial physical model to export the step format feature model;

[0090] Step S606, using a three-dimensional model conversion tool to perform lightweight processing on the step format feature model to obtain the physical three-dimensional model, wherein the physical three-dimensional model is an ive format mesh model.

[0091] It should be noted that in the present embodiment, when training, the corresponding physical entity device three-dimensional model is pre-set on the interactive device, for example, when performing bench operation auxiliary training, the corresponding physical entity device model on the interactive device is a bench operation device three-dimensional model. Therefore, in order to facilitate timely interaction, the physical three-dimensional model needs to be processed. Specifically, first, the initial physical model on the interactive device is obtained and analyzed, and the initial physical model is exported in step format to obtain a step format feature model. Specifically, the initial physical model can be analyzed by using SolidWorks software. Then, the step format feature model is processed by using a three-dimensional model conversion tool to obtain the corresponding physical three-dimensional model in the current application scenario (bench operation device), wherein the physical three-dimensional model is an ive format mesh model. The step format or the ive format is a model saving format, which is known to those skilled in the art and will not be described here. Accordingly, after the above analysis and processing of the three-dimensional model, the actual geometric parameter information of the physical entity device (bench operation device) can be reflected, and the consumption of computing resources can be reduced during the rendering process, thereby ensuring the stable performance of digital interaction.

[0092] Further, in the embodiment of the application, the virtual-real registration based on the virtual three-dimensional model on the physical three-dimensional model specifically includes: matching the coordinates of the virtual three-dimensional model and the physical three-dimensional model based on a preset adjustment method until the geometric position error of the virtual three-dimensional model and the physical three-dimensional model falls within a preset error range; calculating the relative pose matrix of the registered virtual three-dimensional model and the physical three-dimensional model according to the current space six-degree-of-freedom pose, and performing model rework based on the relative pose matrix to complete model virtual-real fusion.

[0093] It should be noted that in the present embodiment, after the virtual three-dimensional model and the physical three-dimensional model are obtained, virtual-real registration is needed to complete the virtual-real fusion of the model, which mainly includes two points, one is coordinate matching, and the other is model coincidence. Essentially, it is to complete one-to-one fusion of the virtual three-dimensional model edited by the user and the existing physical three-dimensional model of the interactive device. Specifically, through functions such as plane fitting, object scaling, and spatial six-degree-of-freedom adjustment, the virtual three-dimensional model and the physical three-dimensional model are registered in position to ensure that the geometric position error of the virtual three-dimensional model and the physical three-dimensional model is not higher than ±10 mm, i.e., the preset error range is ±10 mm. After registration is completed, the current registration data is saved, and then the relative pose matrix of the virtual three-dimensional model and the physical three-dimensional model after registration is calculated according to the current spatial six-degree-of-freedom pose, so as to complete the model registration operation based on the relative pose matrix to complete the virtual-real fusion of the model. It should be noted that the spatial six-degree-of-freedom pose calculation is that the object has six degrees of freedom in space, i.e., the movement freedom along the x, y, and z three orthogonal coordinate axes and the rotation freedom around the three coordinate axes. Therefore, to completely determine the position of the object, the six degrees of freedom must be clear. In the present embodiment, only the application is used, and the specific calculation steps are not described here.

[0094] Further, in an embodiment of the application, the feedback signal on the interactive device is received to complete digital interaction for bench worker training, specifically including: acquiring the feedback signal based on the acquisition device arranged on the interactive device, wherein the acquisition device at least includes an image acquisition device and a microphone; and performing digital interaction on the virtual three-dimensional model based on the feedback signal, wherein the feedback signal at least includes a gesture image signal and a voice operation instruction signal.

[0095] It should be noted that in the present embodiment, after the virtual fusion described above is completed, the new employee (bench worker) is waited for digital interaction under actual operation to assist in training. Specifically, the feedback signal is acquired by the acquisition device arranged on the interactive device. Correspondingly, the acquisition device at least includes an image acquisition device and a microphone. After the feedback signal is obtained, digital interaction is performed on the virtual three-dimensional model based on the feedback signal, wherein the feedback signal at least includes a gesture image signal and a voice operation instruction signal. Through digital interaction, individual training needs of different new employees (bench workers) can be realized, such as realizing operations such as next step, previous step, repeated playing, and selecting detection results in the flowchart to complete overall control of the training process and realize training operation of the bench worker of the steel plant based on the mixed reality technology.

[0096] The embodiment of the application further provides a steel plant maintenance benchwork training device based on a mixed reality technology, which can realize the steel plant maintenance benchwork training method based on the mixed reality technology.

[0097] Please refer to Figure 7 In an embodiment, the embodiment provides a steel plant maintenance benchwork training device 70 based on a mixed reality technology, which comprises:

[0098] A course module 71 is configured to acquire a virtual three-dimensional model, wherein the virtual three-dimensional model comprises course information, and the course information at least comprises benchwork operation flow information and three-dimensional operation action information.

[0099] A device module 72 is configured to acquire a physical three-dimensional model on an interactive device, wherein the physical three-dimensional model comprises a device model of a benchwork operation entity device.

[0100] A registration module 73 is configured to perform virtual-real registration on the physical three-dimensional model based on the virtual three-dimensional model, wherein the virtual-real registration comprises coordinate matching and model coincidence.

[0101] An interactive module 74 is configured to receive a feedback signal on the interactive device to perform digital interaction to complete benchwork training, wherein the interactive process comprises training control on the virtual three-dimensional model.

[0102] Since the specific implementation mode of the embodiment corresponds to the foregoing method embodiment, the same details will not be repeated here, and those skilled in the art should understand that, Figure 7 The division of each module in the embodiment is only a logical function division, and all or part of the modules can be integrated into one or more physical entities in actual implementation, and the modules can all be realized in the form of software calling by a processing element, or all be realized in the form of hardware, or part of the modules be realized in the form of software calling by a processing element and part of the modules be realized in the form of hardware.

[0103] In several embodiments provided by the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the modules / units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules / units can be indirect coupling or communication connection through some interfaces, devices or modules / units, and can be electrical, mechanical or other forms.

[0104] The modules / units described as separate components can or can not be physically separated, and the components shown as modules / units can or can not be physical modules, i.e. can be located in one place or distributed on multiple network units. Part or all of the modules / units can be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in each embodiment of the present application can be integrated in one processing module, or each module / unit can be physically separated, or two or more modules / units can be integrated in one module / unit.

[0105] Those of ordinary skill in the art should further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0106] The embodiments of the present application further provide a computer readable storage medium. Those skilled in the art can understand that all or part of the steps of the methods described in the above embodiments can be instructed by a program to complete the processor, and the program can be stored in the computer readable storage medium. The storage medium is a non-transitory medium, for example, a random access memory, a read only memory, a flash memory, a hard disk, a solid state disk, a magnetic tape, a floppy disk, an optical disc and any combination thereof. The storage medium can be any available medium accessible by a computer or a data storage device such as a server, a data center and the like integrated with one or more available medium sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)) or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0107] The embodiments of the present application can also provide a computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave and the like) mode.

[0108] When the computer program product is executed by a computer, the computer executes the method described in the foregoing method embodiments. The computer program product can be a software installation package, and when the foregoing method is needed, the computer program product can be downloaded and executed on the computer.

[0109] The description of the flow or structure corresponding to each of the above figures has its own emphasis, and the parts not described in detail in a certain flow or structure can be referred to the related description of other flows or structures.

[0110] The embodiments of the present application further provide an electronic device. The electronic device includes a processor and a memory.

[0111] The memory is used to store a computer program.

[0112] The memory includes: ROM, RAM, disk, U disk, memory card or optical disc and various media that can store program codes.

[0113] The processor is connected with the memory, and is used for executing the computer program stored in the memory, so that the electronic equipment executes the steel mill maintenance bench work training method based on the mixed reality technology.

[0114] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP) and the like; and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0115] As shown in Figure 8 The electronic equipment of the present application is in the form of a general-purpose computing device. The components of the electronic equipment can include but are not limited to one or more processors or processing units 81, a memory 82, and a bus 83 connecting different system components, including the memory 82 and the processing unit 81.

[0116] The bus 83 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to an industry standard architecture (ISA) bus, a micro channel architecture (MAC) bus, an enhanced ISA bus, a video electronics standards association (VESA) local bus, and a peripheral component interconnect (PCI) bus.

[0117] The electronic equipment typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic equipment, including volatile and non-volatile media, removable and non-removable media.

[0118] Memory 82 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 821 and / or cache memory 822. The electronic device can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 823 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Figure 8 Although not shown, a magnetic disk drive can also be utilized in some embodiments for reading from and writing to a removable, non-volatile magnetic media such as a "floppy disk," and an optical disk drive can be used for reading from and writing to a removable, non-volatile optical storage media such as an optical disk such as CD-ROM, DVD-ROM or other optical media. In these instances, each drive can be connected to the bus 83 by one or more data media interfaces. The memory 82 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application. Figure 8 Although not shown, a magnetic disk drive can also be utilized in some embodiments for reading from and writing to a removable, non-volatile magnetic media such as a "floppy disk," and an optical disk drive can be used for reading from and writing to a removable, non-volatile optical storage media such as an optical disk such as CD-ROM, DVD-ROM or other optical media. In these instances, each drive can be connected to the bus 83 by one or more data media interfaces. The memory 82 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0119] Program / utility 824 having a set (at least one) of program modules 8241 can be stored in memory 82 by way of example, such as an operating system, one or more application programs, other program modules, and program data, each of which can include an implementation of a network environment, for example. The program modules 8241 typically carry out the functions and / or methodologies of embodiments of the application as described herein.

[0120] The electronic device can also communicate with one or more external devices such as a keyboard or pointing device, a display, etc. via I / O interface 84. Furthermore, the electronic device can communicate with one or more devices that enable a user to interact with the electronic device, for example, a user can interact with the electronic device via a display, keyboard, and / or pointing device. The electronic device can also include a communication interface 86 that can be used to enable communication with one or more other electronic devices. Figure 8 The communication interface 86 can enable communication with one or more other devices, such as a personal computer, a voice communications server, a wireless base station, etc. The communication interface 86 can include any suitable communication interface, such as a wired interface (e.g., an Ethernet interface), a wireless interface, etc. As shown, the communication interface 86 is in communication with the bus 83 via the communication interface 86. It will be appreciated that other hardware and / or software modules that can be used in conjunction with the electronic device can also be utilized, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0121] Referring to Figure 9 In one embodiment, the present application also provides a steel plant maintenance fitter training system 90 based on a mixed reality technology, the system comprising:

[0122] a physical three-dimensional model for the user to model three-dimensionally;

[0123] an interactive device 92, including an MR device, for mediating the user interaction data to digitally interact on the physical three-dimensional model;

[0124] an electronic device 93 as described above, for completing the interactive control of the benchwork training.

[0125] It is to be noted that in the present embodiment, it is specifically described which devices are mainly applied in the training process of the new employee (benchworker), which specifically include the physical device, such as the benchwork operation device described in the above embodiment, the interactive device, which corresponds to the mixed reality technology device, i.e. the MR device, such as the mixed reality glasses, and the electronic device described above, for controlling the digital interaction process of the entire benchwork training. Accordingly, the physical device is used for the user to model three-dimensionally to obtain the physical three-dimensional model in the corresponding scene, and the interactive device is used as the medium between the physical device and the user to complete the relevant digital interaction of the benchwork training on the constructed virtual three-dimensional model under the control of the electronic device.

[0126] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought disclosed by the present application should be covered by the claims of the present application.

Claims

1. A training method for steel plant maintenance fitters based on mixed reality technology, characterized in that, include: A virtual 3D model is obtained, the virtual 3D model including courseware information, the courseware information including at least fitter operation process information and 3D operation action information; Obtain a physical 3D model on the interactive device, wherein the physical 3D model specifically includes a device model of the physical equipment used for fitter operations; Based on the virtual 3D model, virtual-real registration is performed on the physical 3D model, wherein virtual-real registration includes coordinate matching and model overlap; The system receives feedback signals from the interactive device to perform digital interaction in order to complete fitter training. The interaction process includes training control of the virtual 3D model.

2. The training method for steel plant maintenance fitters based on mixed reality technology according to claim 1, characterized in that, The acquisition of the virtual 3D model specifically includes: Obtain editing data from a preset editing tool, wherein the preset editing tool includes a 3D courseware editing tool; Based on the edited data, an operation flow expression is obtained, wherein the expression content includes at least operation nodes, detection nodes, and maintenance nodes; Based on the edited data, node operation data is obtained, wherein the node operation data includes at least the operation content, operation object, tools used, and precautions; The fitter's operation process information is obtained based on the content expressed in the operation process, and the three-dimensional operation action information is obtained based on the node operation data.

3. The training method for steel plant maintenance fitters based on mixed reality technology according to claim 2, characterized in that, The acquisition of the physical 3D model on the interactive device specifically includes: Obtain the initial physical model on the interactive device; The initial physical model is analyzed and the format is exported to obtain the step format feature model; The physical 3D model is obtained by using a 3D model conversion tool to perform lightweight processing on the STEP format feature model, wherein the physical 3D model is an IVE format mesh model.

4. The training method for steel plant maintenance fitters based on mixed reality technology according to claim 2, characterized in that, The method further includes organizing and encrypting the courseware information to form a binary format courseware data file, and storing the courseware data file in a preset location on the interactive device.

5. The training method for steel plant maintenance fitters based on mixed reality technology according to claim 3, characterized in that, The virtual-to-real registration based on the virtual 3D model on the physical 3D model specifically includes: Based on preset adjustment wording, the virtual 3D model and the physical 3D model are matched in coordinates until the geometric position error between the virtual 3D model and the physical 3D model falls within a preset error range; Based on the relative pose matrix of the virtual 3D model and the physical 3D model after registration by the current six-degree-of-freedom pose calculation, model re-coupling is performed based on the relative pose matrix to complete the virtual-real fusion of the model.

6. The training method for steel plant maintenance fitters based on mixed reality technology according to claim 5, characterized in that, The feedback signal received from the interactive device is used for digital interaction to complete fitter training, specifically including: The feedback signal is acquired based on a data acquisition device installed on the interactive device, wherein the data acquisition device includes at least an image acquisition device and a microphone; Digital interaction is performed on the virtual 3D model based on the feedback signal, wherein the feedback signal includes at least gesture image signal and voice operation command signal.

7. A training device for steel plant maintenance fitters based on mixed reality technology, characterized in that, include: The courseware module is used to acquire a virtual 3D model, which includes courseware information, and the courseware information includes at least fitter operation process information and 3D operation action information. The device module is used to acquire a physical 3D model on the interactive device, wherein the physical 3D model specifically includes a device model of the fitter's operation entity device; The registration module is used to perform virtual-real registration on the physical 3D model based on the virtual 3D model, wherein virtual-real registration includes coordinate matching and model overlap; An interaction module is used to receive feedback signals from the interaction device to perform digital interaction in order to complete fitter training. The interaction process includes training control of the virtual 3D model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steel mill maintenance fitter training method based on mixed reality technology as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the steel plant maintenance fitter training method based on mixed reality technology as described in any one of claims 1 to 6.

10. A training system for steel plant maintenance fitters based on mixed reality technology, characterized in that, include: Physical equipment, including fitter's operating equipment, is used by users to perform 3D modeling to obtain physical 3D models; Interactive devices, including MR devices, are used as a medium to digitally interact with user interaction data on the physical three-dimensional model. The electronic device as described in claim 9 is used for interactive control to complete fitter training.

Images