Information equipment management method based on augmented reality technology

Abstract

The invention discloses an informatization equipment management method based on an augmented reality technology in the technical field of informatization equipment management, and aims to solve the problems that in the prior art, the state of target equipment cannot be visually seen, an operator cannot be assisted in reasoning and decision making in real time, maintenance operation is complex, and maintenance cost is high. And the use by non-experienced personnel is inconvenient. The method comprises the following steps: acquiring multi-angle images of informatization equipment and a basic environment, reconstructing the multi-angle images by adopting dense point cloud based on an RMVSNet framework, and performing three-dimensional reconstruction on the informatization equipment and the basic environment to establish a machine room equipment digital model; according to the invention, fault reasons and fault debugging schemes are visually displayed, and interaction between human bodies and a machine room equipment digital model is combined to guide operation of workers and assist the workers in reasoning and decision making, so that the maintenance operation is simplified, use by non-experience personnel is facilitated, and the practicability of the system is ensured.

Description

Technical Field

[0001] The invention relates to an information equipment management method based on augmented reality technology, belonging to the technical field of information equipment management. Background Art

[0002] AR is a technology that promotes the integration of real-world information and virtual-world information content. It is an important technology for realizing the digitalization of the manufacturing process. How to effectively integrate multiple data information such as equipment information, environmental information, and human information, and study the relevant algorithms of AR scenario visualization and troubleshooting decisions are of great significance to improving the overall operation efficiency of the equipment. With the expansion of workshop scale and the increase in the number of equipment, the traditional workshop production method has increasingly shown its shortcomings and limitations. There is an urgent need for workshop production informatization, and AR has also received attention and development. The biggest shortcoming in traditional management is that even if the operation and maintenance personnel have undergone initial training investment, they are unable to make correct operations in actual operations due to the complex production environment; in addition, due to the depth, breadth and time dimension of control, it is difficult for operators to agilely and accurately control equipment stability, equipment quality abnormalities and other aspects in a multi-parameter, multi-disturbance, and strongly correlated virtualized cloud environment.

[0003] In the prior art, when managing information equipment, for example, the patent with publication number CN112162636A discloses an equipment management system and method based on augmented reality and ultra-high frequency reading and writing RFID technology. When the operator is working, the RFID tag is read through the wearable augmented reality glasses to obtain the location information and equipment status information of the target equipment. At the same time, the background server provides instructions or indication information to the wearable augmented reality device. This method cannot intuitively see the status of the target equipment, nor can it assist the operator in reasoning and decision-making in real time. The maintenance operation is relatively complicated and inconvenient for inexperienced personnel to use, which affects the practicality of the method. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide an information equipment management method based on augmented reality technology to solve the problem that the prior art cannot intuitively see the status of the target equipment when managing information equipment, nor can it assist operators in reasoning and decision-making in real time, and the maintenance operation is relatively complicated.

[0005] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions: In a first aspect, the present invention provides an information equipment management method based on augmented reality technology, comprising the following steps: Collect multi-angle images of information equipment and basic environment, reconstruct the multi-angle images using dense point cloud based on RMVSNet framework, and perform 3D reconstruction of information equipment and basic environment to establish a digital model of computer room equipment; Collect images of reference marks in the basic environment, use image recognition algorithms to identify the reference marks, register the corresponding virtual content to specific marking points in the digital model of the equipment room, and obtain the digital model of the equipment room integrated with the basic environment; The dense tracking method is used to realize the dynamic tracking of human skeleton gestures, and the interaction between the human body and the digital model of the equipment in the computer room is realized by combining the digital model of the equipment in the computer room integrated with the basic environment. Obtain the equipment operating status, obtain fault information based on the equipment operating status, substitute the fault information into the pre-built causal decision model, output the fault cause and troubleshooting plan, and guide the staff's operation based on the interaction between the human body and the digital model of the equipment in the computer room to realize the management of information equipment.

[0006] Furthermore, the three-dimensional reconstruction of the information equipment and the basic environment to establish a digital model of the equipment in the computer room specifically includes: Based on multi-angle images of information equipment and basic environment, an optimized unstructured multi-viewpoint 3D reconstruction network deep learning algorithm is used to match the same point in different images, construct feature bodies, aggregate the feature bodies into cost bodies, and then regularize them through recurrent neural networks to obtain feature maps. Convolution and GNU are used to process each feature map to construct a 3D point cloud and form a digital model of the computer room equipment.

[0007] Furthermore, the step of obtaining a digital model of the equipment room integrated with the basic environment further includes: Using situational visualization technology, the benchmark markers in the basic environment interact with specific marker points in the digital model of the equipment room equipment.

[0008] Furthermore, the step of obtaining a digital model of the equipment room integrated with the basic environment further includes: Analyze the saliency of images of fiducial markers in a base environment; An image search algorithm is applied to search for regions in the image of the fiducial markers in the base environment that do not have these features based on the fiducial markers' features, and to place labels at less salient locations.

[0009] Furthermore, the construction of the causal decision model includes: Obtain expert experience, use the expert experience as basic data, digitally attribute faults, and establish a causal decision-making model.

[0010] Furthermore, the output fault cause and fault troubleshooting solution also include: The Bayesian network is used to learn more potential fault data from the original data. Based on the potential fault data, an auxiliary model is established to assist the causal decision model in outputting the fault cause and fault troubleshooting plan.

[0011] Furthermore, the original data includes parameters of information equipment.

[0012] In a second aspect, the present invention provides an information equipment management device based on augmented reality technology, comprising: Acquisition module: collects multi-angle images of information equipment and basic environment, reconstructs the multi-angle images using dense point clouds based on the RMVSNet framework, and performs three-dimensional reconstruction of information equipment and basic environment to establish a digital model of equipment in the computer room; Interaction module: collect images of reference marks in the basic environment, use image recognition algorithms to identify the reference marks, register the corresponding virtual content to specific marking points in the digital model of the equipment in the computer room, and obtain the digital model of the equipment in the computer room integrated with the basic environment; use dense tracking methods to achieve dynamic skeletal tracking of human gestures, and combine the digital model of the equipment in the computer room integrated with the basic environment to achieve interaction between the human body and the digital model of the equipment in the computer room; Guidance module: obtains the equipment operation status, obtains fault information based on the equipment operation status, substitutes the fault information into the pre-built causal decision model, outputs the fault cause and troubleshooting plan, and guides the staff to operate based on the interaction between the human body and the digital model of the equipment in the computer room to realize the management of information equipment.

[0013] In a third aspect, the present invention provides a terminal, including a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to the first aspect.

[0014] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The information equipment management method based on augmented reality technology first constructs a digital model of computer room equipment based on information equipment and basic environment, then realizes scene visualization through AR scene visualization image recognition algorithm, and finally establishes a causal decision model of computer room equipment failure to realize intelligent failure decision-making, intuitively display the cause of the failure and the troubleshooting plan, and combines the interaction between the human body and the digital model of the computer room equipment to guide the staff's operation, assist the staff in reasoning and decision-making, simplify the maintenance operation, facilitate the use of inexperienced personnel, and ensure the practicality of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of an information equipment management method based on augmented reality technology provided according to an embodiment of the present invention; Figure 2 is a specific workflow diagram of an information equipment management method based on augmented reality technology provided according to an embodiment of the present invention; Figure 3 Schematic diagram of geometric constraints of three-dimensional reconstruction matching points provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0018] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0019] Embodiment 1:

[0020] like Figure 1-Figure 2 As shown, the present invention provides an information equipment management method based on augmented reality technology, comprising the following steps: Collect multi-angle images of information equipment and basic environment, reconstruct the multi-angle images using dense point cloud based on RMVSNet framework, and perform 3D reconstruction of information equipment and basic environment to establish a digital model of computer room equipment; Collect images of reference marks in the basic environment, use image recognition algorithms to identify the reference marks, register the corresponding virtual content to specific marking points in the digital model of the equipment room, and obtain the digital model of the equipment room integrated with the basic environment; The dense tracking method is used to realize the dynamic tracking of human skeleton gestures. Combined with the digital model of the computer room equipment integrated with the basic environment, the interaction between the human body and the digital model of the computer room equipment is realized, that is, AR scene visualization. Obtain the equipment operating status, obtain fault information based on the equipment operating status, substitute the fault information into the pre-built causal decision model, output the fault cause and troubleshooting plan, and guide the staff's operation based on the interaction between the human body and the digital model of the equipment in the computer room to realize the management of information equipment.

[0021] In this embodiment, the three-dimensional reconstruction of the information equipment and the basic environment to establish a digital model of the equipment in the computer room specifically includes: Based on multi-angle images of information equipment and basic environment, an optimized unstructured multi-view 3D reconstruction network deep learning algorithm is used to match the same point in different images (such as Figure 3 As shown in the figure), construct feature bodies, aggregate the feature bodies into cost bodies, and then regularize them through a recurrent neural network to obtain feature maps. Use convolution and GNU to process each feature map to construct a three-dimensional point cloud and form a digital model of the equipment in the computer room.

[0022] In this embodiment, obtaining a digital model of the equipment room equipment integrated with the basic environment further includes: By using situational visualization technology, the benchmark markers in the basic environment can interact with the specific marking points in the digital model of the computer room equipment, reducing the mismatch in the process of combining virtual and real, and improving information fusion technology.

[0023] In one embodiment, the step of obtaining a digital model of equipment in a computer room integrated with a basic environment further includes: Analyze the saliency of images of fiducial markers in a base environment; An image search algorithm is applied to search for areas in the image of the fiducial markers in the base environment that do not have these features based on the features of the fiducial markers, and the labels are placed at positions with lower significance, thereby avoiding the labels from covering heavy fiducial markers in the base environment; optionally, the labels include information such as the parameters and name of the device.

[0024] In one embodiment, the construction of the causal decision model includes: Obtain expert experience, use the expert experience as basic data, digitally attribute faults, and establish a causal decision-making model.

[0025] In one embodiment, the output fault cause and fault troubleshooting solution further include: A Bayesian network is used to learn more potential fault data from the original data, and an auxiliary model is established based on the potential fault data to assist the causal decision model in outputting the fault cause and fault troubleshooting plan; optionally, the original data includes parameters of the information equipment.

[0026] The present invention takes the AR algorithm of equipment management as the research core, explores the digital model of the entire life cycle of information equipment, introduces three-dimensional reconstruction technology, and uses the image recognition algorithm of AR scene visualization to perceive basic environmental information and human body information, detect on-site faults, guide the on-site operation of operation and maintenance personnel, and improve the quality of operation and maintenance; introduces previously collected and stored equipment data (i.e., expert experience) as basic information, constructs a digital expert decision-making algorithm, and quickly proposes a fault elimination plan based on the decision-making model, shortens the maintenance time, and reduces losses. This method is of great significance to the in-depth development of the current "cost reduction, quality improvement, and efficiency improvement" work.

[0027] The present invention firstly constructs a digital model of computer room equipment based on information equipment and basic environment, then realizes scene visualization through AR scene visualization image recognition algorithm, and finally establishes a causal decision model of computer room equipment failure, realizes intelligent failure decision-making, intuitively displays the cause of the failure and the troubleshooting plan, and combines the interaction between the human body and the digital model of the computer room equipment to guide the staff's operation, assist the staff in reasoning and decision-making, simplify the maintenance operation, facilitate the use of inexperienced personnel, and ensure the practicality of the present invention.

[0028] Embodiment 2:

[0029] The present invention provides an information equipment management device based on augmented reality technology, comprising: Acquisition module: collects multi-angle images of information equipment and basic environment, reconstructs the multi-angle images using dense point clouds based on the RMVSNet framework, and performs three-dimensional reconstruction of information equipment and basic environment to establish a digital model of equipment in the computer room; Interaction module: collect images of reference marks in the basic environment, use image recognition algorithms to identify the reference marks, register the corresponding virtual content to specific marking points in the digital model of the equipment in the computer room, and obtain the digital model of the equipment in the computer room integrated with the basic environment; use dense tracking methods to achieve dynamic skeletal tracking of human gestures, and combine the digital model of the equipment in the computer room integrated with the basic environment to achieve interaction between the human body and the digital model of the equipment in the computer room; Guidance module: obtains the equipment operation status, obtains fault information based on the equipment operation status, substitutes the fault information into the pre-built causal decision model, outputs the fault cause and troubleshooting plan, and guides the staff to operate based on the interaction between the human body and the digital model of the equipment in the computer room to realize the management of information equipment.

[0030] Optional, such as Figure 3 As shown, the present invention collects multi-angle images of information equipment and basic environment through cameras, and the cameras include camera a and camera b.

[0031] Embodiment three:

[0032] An embodiment of the present invention further provides a terminal, including a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method described in embodiment 1.

[0033] Embodiment 4:

[0034] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method described in the first embodiment are implemented.

[0035] Since the storage medium provided in the embodiment of the present invention can execute the method provided in the first embodiment of the present invention, it has the corresponding functional modules and beneficial effects of the execution method.

[0036] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0037] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0038] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0039] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0040] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An information equipment management method based on augmented reality technology, characterized in that: The following steps are involved: Collect multi-angle images of information equipment and basic environment, reconstruct the multi-angle images using dense point cloud based on RMVSNet framework, and perform 3D reconstruction of information equipment and basic environment to establish a digital model of computer room equipment; Collect images of reference marks in the basic environment, use image recognition algorithms to identify the reference marks, register the corresponding virtual content to specific marking points in the digital model of the equipment room, and obtain the digital model of the equipment room integrated with the basic environment; The dense tracking method is used to realize the dynamic tracking of human skeleton gestures, and the interaction between the human body and the digital model of the equipment in the computer room is realized by combining the digital model of the equipment in the computer room integrated with the basic environment. Obtain the equipment operating status, obtain fault information based on the equipment operating status, substitute the fault information into the pre-built causal decision model, output the fault cause and troubleshooting plan, and guide the staff's operation based on the interaction between the human body and the digital model of the equipment in the computer room to realize the management of information equipment.

2. The information equipment management method based on augmented reality technology according to claim 1 is characterized in that: The three-dimensional reconstruction of the information equipment and basic environment to establish a digital model of the equipment in the computer room specifically includes: Based on multi-angle images of information equipment and basic environment, an optimized unstructured multi-viewpoint 3D reconstruction network deep learning algorithm is used to match the same point in different images, construct feature bodies, aggregate the feature bodies into cost bodies, and then regularize them through recurrent neural networks to obtain feature maps. Convolution and GNU are used to process each feature map to construct a 3D point cloud and form a digital model of the computer room equipment.

3. The information equipment management method based on augmented reality technology according to claim 1 is characterized in that: The obtaining of the digital model of the equipment room integrated with the basic environment also includes: Using situational visualization technology, the benchmark markers in the basic environment interact with specific marker points in the digital model of the equipment room equipment.

4. The information equipment management method based on augmented reality technology according to claim 1 is characterized in that: The obtaining of the digital model of the equipment room integrated with the basic environment also includes: Analyze the saliency of images of fiducial markers in a base environment; An image search algorithm is applied to search for regions in the image of the fiducial markers in the base environment that do not have these features based on the fiducial markers' features, and to place labels at less salient locations.

5. The information equipment management method based on augmented reality technology according to claim 1 is characterized in that: The construction of the causal decision model includes: Obtain expert experience, use the expert experience as basic data, digitally attribute faults, and establish a causal decision-making model.

6. The information equipment management method based on augmented reality technology according to claim 1 is characterized in that: The output fault causes and troubleshooting solutions also include: The Bayesian network is used to learn more potential fault data from the original data. Based on the potential fault data, an auxiliary model is established to assist the causal decision model in outputting the fault cause and fault troubleshooting plan.

7. The information equipment management method based on augmented reality technology according to claim 6 is characterized in that: The raw data include parameters of information equipment.

8. An information equipment management device based on augmented reality technology, characterized in that: include: Acquisition module: collects multi-angle images of information equipment and basic environment, reconstructs the multi-angle images using dense point clouds based on the RMVSNet framework, and performs three-dimensional reconstruction of information equipment and basic environment to establish a digital model of equipment in the computer room; Interaction module: collect images of reference marks in the basic environment, use image recognition algorithms to identify the reference marks, register the corresponding virtual content to specific marking points in the digital model of the equipment in the computer room, and obtain the digital model of the equipment in the computer room integrated with the basic environment; use dense tracking methods to achieve dynamic skeletal tracking of human gestures, and combine the digital model of the equipment in the computer room integrated with the basic environment to achieve interaction between the human body and the digital model of the equipment in the computer room; Guidance module: obtains the equipment operation status, obtains fault information based on the equipment operation status, substitutes the fault information into the pre-built causal decision model, outputs the fault cause and troubleshooting plan, and guides the staff to operate based on the interaction between the human body and the digital model of the equipment in the computer room to realize the management of information equipment.

9. A terminal, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images