An augmented reality-based intelligent assembly and operation and maintenance guidance method, system and device and storage medium

Abstract

The application discloses an intelligent assembly and operation and maintenance guidance method, system and device based on augmented reality and a storage medium, and belongs to the technical field of equipment remote assembly and operation and maintenance, and the method comprises the following steps: collecting assembly and operation and maintenance pictures and corresponding assembly and operation and maintenance guidance information, inputting the pictures and the guidance information into a constructed assembly and operation and maintenance model, generating a database, and collecting a real scene; performing visual processing on the assembly and operation and maintenance scene in the database through computer vision, and identifying an assembly and operation and maintenance target; performing virtual-real fusion on the real scene after solving the registration pose of a virtual guidance tool according to the assembly and operation and maintenance target, and obtaining an augmented reality assembly and operation and maintenance scene; and displaying the augmented reality assembly and operation and maintenance scene to an operator for assembly and operation and maintenance guidance; the application can reduce work intensity, avoid slight damage to equipment, reduce economic and all-around losses caused by equipment damage, improve the reliability of assembly and operation and maintenance of complex equipment, and improve the controllable degree of operation quality.

Description

Technical Field

[0001] This invention relates to an intelligent assembly and maintenance guidance method, system, device, and storage medium based on augmented reality, belonging to the field of remote equipment assembly and maintenance technology. Background Technology

[0002] Radar electronic equipment is the core equipment for realizing intelligent perception of national physical space security. Radar electronic equipment is rapidly developing towards high density and high integration. Currently, the trend of miniaturization, high power, and multi-functional integration of high-density radar equipment is becoming increasingly obvious. With the increasing performance requirements of radar equipment, there are more and more internal devices, higher density requirements, and smaller operating spaces. The requirements for assembly and maintenance are becoming more and more stringent. Especially when maintaining normally functioning radar equipment in a confined space, the harsh heat, air, noise, vibration environment, and electromagnetic radiation put great pressure on human physiology and psychology, and may even cause irreversible effects. This problem of high work intensity and uncontrollable work quality has brought unprecedented challenges to the assembly and maintenance of high-density radar electronic equipment. Summary of the Invention

[0003] The purpose of this invention is to provide an intelligent assembly and maintenance guidance method, system, device and storage medium based on augmented reality, which solves the problems of high workload and uncontrollable work quality in the prior art.

[0004] To achieve the above objectives, the present invention employs the following technical solution:

[0005] In a first aspect, the present invention provides an intelligent assembly and maintenance guidance method based on augmented reality, comprising:

[0006] Collect assembly and maintenance images and corresponding assembly and maintenance guidance information, input them into the constructed assembly and maintenance model, generate a database, and collect real-world scenarios;

[0007] By using computer vision to perform visual processing on assembly and maintenance scenarios in the database, assembly and maintenance targets can be identified.

[0008] After solving the registration pose of the virtual guidance tool based on the assembly and maintenance goals, the virtual and real scenes are fused together to obtain the augmented reality assembly and maintenance scene.

[0009] Augmented reality assembly and maintenance scenarios are displayed to operators to provide assembly and maintenance guidance.

[0010] In conjunction with the first aspect, the database further includes information on the entire assembly process and operation and maintenance scenarios.

[0011] In conjunction with the first aspect, the assembly and maintenance guidance information further includes the operation steps, tool information, and equipment name of the assembly and maintenance process.

[0012] In conjunction with the first aspect, the visual processing further involves extracting typical features from the assembly and maintenance scene using the RGB-D camera in the Microsoft HoloLens augmented glasses, including the identification and tracking of assembly and maintenance targets, 3D scene reconstruction, and computer pose calculation.

[0013] In addition to the first aspect, augmented reality assembly and maintenance scenarios are further displayed to operators through head-mounted displays to provide guidance.

[0014] In addition to the first aspect, it further includes the steps of operators controlling the virtual guidance tool through voice or physical gestures.

[0015] Secondly, the present invention also provides an augmented reality-based intelligent assembly and maintenance guidance system, comprising:

[0016] The data acquisition module is used to collect assembly and maintenance images and corresponding assembly and maintenance guidance information, input them into the constructed assembly and maintenance model, generate a database, and collect real-world scenarios.

[0017] The assembly and maintenance object visual processing module is used to perform visual processing on the assembly and maintenance scenarios in the database through computer vision, and to identify the assembly and maintenance targets.

[0018] The virtual assembly and maintenance tool registration fusion module is used to solve the registration pose of the virtual guidance tool according to the assembly and maintenance goals, and then perform virtual-real fusion with the real scene to obtain the augmented reality assembly and maintenance scene.

[0019] The human-computer interaction module is used to display augmented reality assembly and maintenance scenarios to operators for assembly and maintenance guidance. Operators can then use this module to control the virtual guidance tool.

[0020] In conjunction with the second aspect, the human-computer interaction module is further used to provide functions such as structural analysis, problem detection, and learning and training of physical objects by combining augmented reality technology with AI.

[0021] Thirdly, the present invention also provides an intelligent assembly and maintenance guidance device based on augmented reality, including a processor and a storage medium;

[0022] The storage medium is used to store instructions;

[0023] The processor is configured to operate according to the instructions to perform the steps of the method according to any one of the first aspects.

[0024] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any of the first aspects.

[0025] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0026] This invention provides an augmented reality-based intelligent assembly and maintenance guidance method, system, device, and storage medium. By inputting assembly and maintenance images and corresponding guidance information into a pre-constructed assembly and maintenance model, a database is created for comparison and retrieval during online assembly and maintenance. Computer vision is used to visually process the assembly and maintenance scenes in the database, identifying the assembly and maintenance targets and solving the registration pose of the virtual guidance tool. This is then fused with the real-world scene to obtain an augmented reality assembly and maintenance scene, which is then presented to staff for assembly and maintenance guidance. This allows staff to remotely perform virtual assembly and maintenance of complex equipment, reducing workload, avoiding minor damage to equipment during actual assembly and maintenance, mitigating economic and other losses caused by equipment damage, improving the reliability of complex equipment assembly and maintenance, and enhancing the controllability of work quality. Attached Figure Description

[0027] Figure 1 This is a flowchart of an intelligent assembly and maintenance guidance method based on augmented reality provided in an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of an intelligent assembly and maintenance guidance system based on augmented reality provided in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to illustrate the technical solution of the present invention more clearly, and should not be used to limit the scope of protection of the present invention.

[0030] Example 1

[0031] like Figure 1 As shown in the figure, an embodiment of the present invention provides an intelligent assembly and maintenance guidance method based on augmented reality, which includes the following steps:

[0032] S1. Collect assembly and maintenance images and corresponding assembly and maintenance guidance information, input them into the constructed assembly and maintenance model, generate a database, and collect real-world scenarios.

[0033] Before online assembly and maintenance begins, the functional process of augmented reality-guided assembly and maintenance is modeled to obtain an assembly and maintenance model. By collecting relevant assembly and maintenance images and corresponding assembly and maintenance guidance information, these are input into the constructed assembly and maintenance model, thereby generating a corresponding database for comparison and use during the online assembly and maintenance phase, and for collecting real-world scenarios.

[0034] The database contains information on the entire assembly process and operation and maintenance scenarios. The assembly and operation and maintenance guidance information includes the operation steps, tool information, and equipment names for the assembly and operation and maintenance process.

[0035] S2. Use computer vision to perform visual processing on the assembly and maintenance scenarios in the database to identify the assembly and maintenance targets.

[0036] Computer vision technology is used to perform visual processing on assembly and maintenance scenarios in the database. The visual processing is achieved by extracting typical features of the assembly and maintenance scenarios using the RGB-D camera in the Microsoft HoloLens augmented glasses. This includes the identification and tracking of assembly and maintenance targets, 3D scene reconstruction, and computer pose calculation. The identification and tracking of assembly and maintenance targets (i.e., equipment to be assembled or maintained) is the key part of this process.

[0037] S3. After solving the registration pose of the virtual guidance tool according to the assembly and maintenance goals, the virtual and real scenes are fused together to obtain the augmented reality assembly and maintenance scene.

[0038] After obtaining the registration pose of the virtual guide tool, it is fused with the real scene collected in step S1 to achieve seamless integration.

[0039] S4. Show augmented reality assembly and maintenance scenarios to the operators to provide assembly and maintenance guidance.

[0040] Augmented reality assembly and maintenance scenarios are displayed to operators through head-mounted displays to provide guidance.

[0041] It also includes steps for operators to control the virtual guidance tool through voice or physical gestures.

[0042] The method described in this invention utilizes Microsoft HoloLens augmented reality glasses, which are equipped with both RGB and depth cameras. Compared to traditional single RGB detection or grayscale detection, the RGB-D-based joint detection method integrates depth information of assembly and maintenance targets, effectively improving the speed, accuracy, and robustness of assembly and maintenance target recognition and scene reconstruction. This enhances the quality of equipment assembly and maintenance, shortens cycles, reduces costs, improves worker comfort and safety, and significantly improves the overall assembly and maintenance capabilities of high-density electronic equipment, thus promoting a shift in manufacturing models.

[0043] Example 2

[0044] A complete assembly process can be divided into two stages: the assembly planning stage and the assembly implementation stage. These two stages are usually handled by different departments. First, professionals plan the entire assembly process, and then the assembly line assembles the workpieces according to the planned assembly sequence.

[0045] The embodiments of the present invention propose, as follows: Figure 2 The illustrated intelligent assembly and maintenance guidance system based on augmented reality integrates planning, training, and guidance into a single system, thereby improving the efficiency of the integration between the training and actual operation phases. This includes:

[0046] The data acquisition module is used to collect assembly and maintenance images and corresponding assembly and maintenance guidance information, input them into the constructed assembly and maintenance model, generate a database, and collect real-world scenarios.

[0047] The assembly and maintenance object visual processing module is used to perform visual processing on the assembly and maintenance scenarios in the database through computer vision, and to identify the assembly and maintenance targets.

[0048] The virtual assembly and maintenance tool registration fusion module is used to solve the registration pose of the virtual guidance tool according to the assembly and maintenance goals, and then perform virtual-real fusion with the real scene to obtain the augmented reality assembly and maintenance scene.

[0049] The human-computer interaction module is used to display augmented reality assembly and maintenance scenarios to operators for assembly and maintenance guidance. Operators can then use this module to control the virtual guidance tool.

[0050] The human-computer interaction module is also used to provide functions such as structural analysis, problem detection, and learning and training of physical objects by combining augmented reality technology with AI.

[0051] Example 3

[0052] An embodiment of the present invention provides an intelligent assembly and maintenance guidance device based on augmented reality, comprising a processor and a storage medium;

[0053] The storage medium is used to store instructions;

[0054] The processor is configured to operate according to the instructions to perform the steps of the following method:

[0055] Collect assembly and maintenance images and corresponding assembly and maintenance guidance information, input them into the constructed assembly and maintenance model, generate a database, and collect real-world scenarios;

[0056] By using computer vision to perform visual processing on assembly and maintenance scenarios in the database, assembly and maintenance targets can be identified.

[0057] After solving the registration pose of the virtual guidance tool based on the assembly and maintenance goals, the virtual and real scenes are fused together to obtain the augmented reality assembly and maintenance scene.

[0058] Augmented reality assembly and maintenance scenarios are displayed to operators to provide assembly and maintenance guidance.

[0059] Example 4

[0060] The computer-readable storage medium provided in this embodiment of the invention stores a computer program thereon, characterized in that, when executed by a processor, the program implements the steps of the following method:

[0061] Collect assembly and maintenance images and corresponding assembly and maintenance guidance information, input them into the constructed assembly and maintenance model, generate a database, and collect real-world scenarios;

[0062] By using computer vision to perform visual processing on assembly and maintenance scenarios in the database, assembly and maintenance targets can be identified.

[0063] After solving the registration pose of the virtual guidance tool based on the assembly and maintenance goals, the virtual and real scenes are fused together to obtain the augmented reality assembly and maintenance scene.

[0064] Augmented reality assembly and maintenance scenarios are displayed to operators to provide assembly and maintenance guidance.

[0065] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0067] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0068] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0069] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent assembly and operation and maintenance guidance based on augmented reality, characterized in that, include: Collect assembly and maintenance images and corresponding assembly and maintenance guidance information, input them into the constructed assembly and maintenance model, generate a database, and collect real-world scenarios; By using computer vision to perform visual processing on assembly and maintenance scenarios in the database, assembly and maintenance targets can be identified. After solving the registration pose of the virtual guidance tool based on the assembly and maintenance goals, the virtual and real scenes are fused together to obtain the augmented reality assembly and maintenance scene. Augmented reality assembly and maintenance scenarios are displayed to operators to provide assembly and maintenance guidance; The database contains information on the entire assembly process and operation and maintenance scenarios; The assembly and maintenance guidance information includes the operation steps, tool information, and equipment name for the assembly and maintenance process; The visual processing involves extracting typical features from the assembly and maintenance scene using the RGB-D camera in the Microsoft HoloLens augmented glasses, including the identification and tracking of assembly and maintenance targets, 3D scene reconstruction, and computer pose calculation. Augmented reality assembly and maintenance scenarios are displayed to operators through head-mounted displays to provide guidance; It also includes steps for operators to control the virtual guidance tool through voice or physical gestures.

2. An augmented reality-based intelligent assembly and maintenance guidance system based on the method of claim 1, characterized in that, include: The data acquisition module is used to collect assembly and maintenance images and corresponding assembly and maintenance guidance information, input them into the constructed assembly and maintenance model, generate a database, and collect real-world scenarios. The assembly and maintenance object visual processing module is used to perform visual processing on the assembly and maintenance scenarios in the database through computer vision, and to identify the assembly and maintenance targets. The virtual assembly and maintenance tool registration fusion module is used to solve the registration pose of the virtual guidance tool according to the assembly and maintenance goals, and then perform virtual-real fusion with the real scene to obtain the augmented reality assembly and maintenance scene. The human-computer interaction module is used to display augmented reality assembly and maintenance scenarios to operators for assembly and maintenance guidance. Operators can then use this module to control the virtual guidance tool.

3. The intelligent assembly and maintenance guidance system based on augmented reality according to claim 2, characterized in that, The human-computer interaction module is also used to provide functions such as structural analysis, problem detection, and learning and training of physical objects by combining augmented reality technology with AI.

4. An intelligent assembly and maintenance guidance device based on augmented reality, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to claim 1.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method of claim 1.

Images