Repair and maintenance method, system and equipment based on AR glasses and storage medium

Abstract

The invention relates to a repair and maintenance method, system and device based on AR glasses and a storage medium. The repair and maintenance method comprises the following steps: acquiring information of a to-be-detected device; wherein the information of the to-be-detected equipment comprises an image of the to-be-detected equipment, the position of the to-be-detected equipment and maintenance guidance information of the to-be-detected equipment; constructing and detecting three-dimensional virtual detection equipment based on the to-be-detected equipment image; obtaining a display position in a virtual interface based on the position of the camera and the position of the detection three-dimensional virtual detection equipment; displaying the maintenance guidance information of the to-be-detected equipment and the detection three-dimensional virtual detection equipment on a display position in a virtual interface in an overlapping manner, and generating a maintenance guidance image which is displayed by taking the to-be-maintained equipment as a center; through an advanced display technology and a graphic processing technology, maintenance guidance information is overlaid in a visual field of a user in an augmented reality form, and is combined with an actual equipment image, so that a visual guidance function is realized.

Description

Technical Field

[0001] The present invention relates to the field of AR glasses application technology, and in particular to an inspection and maintenance method, system, computer equipment and storage medium based on AR glasses. Background Art

[0002] In the industrial field, equipment inspection and maintenance is an important means to ensure production safety, improve efficiency and extend equipment life. Traditional inspection and maintenance methods usually rely on manuals and maintenance instructions in paper or electronic format. Workers need to constantly refer to these materials during operation, which not only takes up both hands, but may also lead to inefficiency and incorrect maintenance steps. In addition, due to the continuous upgrading of the structure and functions of complex equipment, maintenance personnel often need to constantly learn new relevant knowledge, which invisibly increases the workload of technicians. Summary of the invention

[0003] The main purpose of the present invention is to provide a maintenance method, system, computer equipment and storage medium based on AR glasses. Through advanced display technology and graphics processing technology, maintenance guidance information is superimposed on the user's field of view in the form of augmented reality and combined with actual equipment images to achieve intuitive guidance functions.

[0004] To achieve the above object, the present invention provides an inspection and maintenance method based on AR glasses, wherein the AR glasses include a virtual interface and a camera, and the inspection and maintenance method includes: Acquire information of the device to be detected; wherein the information of the device to be detected includes an image of the device to be detected, a location of the device to be detected, and maintenance guidance information and detection work tasks of the device to be detected; Constructing a three-dimensional virtual testing device based on the image of the device to be tested; Based on the position of the camera and the position of the device to be detected, obtaining a display position in a virtual interface; The maintenance guidance information of the device to be inspected and the three-dimensional virtual testing device are superimposed and displayed at a display position in the virtual interface to generate a maintenance guidance image centered on the device to be inspected.

[0005] Furthermore, the step of obtaining the information of the device to be detected includes: A working QR code is set on the device to be detected, and information task extraction is performed on the working QR code to obtain extraction task information; The extraction task information is input into the AR glasses to obtain the detection work task; wherein the detection work task includes an operation process for maintaining the equipment to be detected.

[0006] Furthermore, obtaining the display position in the virtual interface based on the position of the camera and the position of the device to be detected includes: Obtaining the position of the three-dimensional virtual testing device based on the position of the device to be tested; Calculating the relative position between the camera and the three-dimensional virtual measuring device to obtain an initial relative distance value between the two; Based on the initial relative distance value, a display position for displaying the image is determined in the virtual interface.

[0007] Furthermore, the maintenance guidance information of the device to be inspected and the three-dimensional virtual device to be inspected are superimposed and displayed at a display position in the virtual interface to generate a maintenance guidance image centered on the device to be inspected, including: Inputting the maintenance guidance information of the equipment to be inspected and the three-dimensional virtual inspection equipment into a preset computer vision model for fusion to obtain a fused image; Obtaining, for the three-dimensional virtual testing device to be tested, an operation and maintenance period of the three-dimensional virtual testing device to be tested; Performing task analysis on the three-dimensional virtual testing equipment to be tested based on the operation and maintenance cycle to obtain the testing work task, and obtaining the AR work task and AR work task item based on the testing work task; Based on the AR operation task and AR operation task item, the position of the fused image is adjusted by a preset dynamic line of sight tracking adjustment method to obtain a target image; wherein the target image is a maintenance guidance image displayed with the equipment to be repaired as the center.

[0008] Furthermore, it also includes: Acquire operation information of maintenance personnel; wherein the operation information is information that maintenance personnel need to follow specific steps to perform operations when performing equipment inspection and maintenance; Extracting the operation information to obtain feedback information from the operator; Input the feedback information into a preset data processing model for data processing and analysis to obtain a maintenance form for the equipment to be tested; Encrypting the maintenance form of the device to be detected to obtain an encrypted maintenance table, and storing the encrypted maintenance table; Alternatively, after the step of obtaining the operation information of the maintenance personnel, the method further comprises: Extracting operation information from the operation information to obtain operation feature information; The operation characteristic information is input into a preset assessment analysis model for analysis or the target personnel remotely view the operation characteristic information of the operation operation background to obtain the analysis result of the assessment; wherein the assessment analysis model includes the pre-input evaluation rules; Obtaining an assessment report based on the analysis results of the assessment; An operation performance calculation is performed based on the evaluation report to obtain the operation performance of the maintenance personnel.

[0009] Furthermore, it also includes: When the equipment to be inspected is found to have defects, a defect document is generated for the equipment to be inspected; Acquire defect problem information inquired by maintenance personnel based on the defect document; Input the defect problem information of the maintenance personnel into a preset maintenance information knowledge base for information matching or remotely inquire the target personnel to obtain the answer corresponding to the inquiry question information of the maintenance personnel; Analyze the maintenance personnel's inquiry information to obtain answers related to the maintenance personnel's inquiry information; Processing the inquiry information of the maintenance personnel to obtain deeper question information of the inquiry information of the maintenance personnel; The deeper question information of the maintenance personnel's inquiry information and the answers associated with the inquiry question information are fused to obtain fused information, and the fused information is added to the inspection and maintenance information knowledge base.

[0010] Furthermore, the three-dimensional virtual testing device is constructed based on the image of the device to be tested, including: Taking images of the device to be detected to obtain images of the device to be detected from different viewing angles; The point cloud data of the image of the device to be detected at different viewing angles is obtained by using a preset SLAM technology; wherein the point cloud data is a data set composed of three-dimensional point coordinates on the image of the device to be detected at different viewing angles; Based on the point cloud data, construct a three-dimensional spatial model of the device to be detected; Preprocessing the three-dimensional space model of the device to be detected to obtain a target three-dimensional space model of the device to be detected; Adding annotation information to the target three-dimensional space model through a preset annotation tool to obtain a three-dimensional space model with annotations; wherein the annotation information includes the fault location, maintenance steps and parts replacement instructions of the equipment to be detected; Rendering the annotated three-dimensional space model to obtain a rendered three-dimensional space model, and using the rendered three-dimensional space model as a three-dimensional virtual testing device to be tested; wherein the rendered three-dimensional space model is used for visual display; The three-dimensional virtual testing device to be tested is stored in a device library.

[0011] In addition, the present application also provides an inspection and maintenance system based on AR glasses, wherein the AR glasses include a virtual interface and a camera, and the inspection and maintenance system includes: An acquisition module is used to acquire information of the device to be detected; wherein the information of the device to be detected includes an image of the device to be detected, a location of the device to be detected, and maintenance guidance information and detection work tasks of the device to be detected; A construction module, used for constructing a three-dimensional virtual testing device based on the image of the device to be tested; A display module, used for obtaining a display position in a virtual interface based on the position of the camera and the position of the three-dimensional virtual measuring device; The superposition module is used to superimpose the maintenance guidance information of the device to be inspected and the three-dimensional virtual test device on a display position in the virtual interface to generate a maintenance guidance image centered on the device to be inspected.

[0012] In addition, the present application also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and wherein the processor implements the steps of the above method when executing the computer program.

[0013] In addition, the present application also provides a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described above are implemented.

[0014] The AR glasses-based maintenance method, system, computer equipment and storage medium provided by the present invention include the following steps: obtaining information of a device to be detected; wherein the information of the device to be detected includes an image of the device to be detected, a position of the device to be detected and maintenance guidance information of the device to be detected; constructing a three-dimensional virtual testing device for detection based on the image of the device to be detected; obtaining a display position in a virtual interface based on the position of the camera and the position of the three-dimensional virtual testing device for detection; superimposing the maintenance guidance information of the device to be detected and the three-dimensional virtual testing device for detection on the display position in the virtual interface and displaying them, generating a maintenance guidance image centered on the device to be detected; directly displaying correct maintenance steps and precautions as well as possible warning information in real time within the staff's field of vision through the virtual interface, thereby reducing operational errors and potential safety risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the steps of a repair and maintenance method based on AR glasses in one embodiment of the present invention; Figure 2 It is a structural block diagram of a repair and maintenance system based on AR glasses in one embodiment of the present invention; Figure 3It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0016] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] like Figure 1 As shown, Figure 1 It is a schematic diagram of the steps of a repair and maintenance method based on AR glasses in one embodiment of the present invention; In one embodiment of the present invention, a maintenance method based on AR glasses is provided. The AR glasses include a virtual interface and a camera. The method includes the following steps: Step S1, obtaining information of the device to be detected; wherein the information of the device to be detected includes an image of the device to be detected, a location of the device to be detected, and maintenance guidance information and detection work tasks of the device to be detected.

[0019] Specifically, use a high-resolution camera or camera to capture images of the equipment to be inspected. If conditions permit, use drones, robots or other automated equipment to capture images to obtain a more comprehensive perspective. Obtain the precise geographic location information of the equipment through GPS devices. If the equipment is located indoors or in an environment with weak GPS signals, an indoor positioning system can be used, such as positioning technology based on Wi-Fi, Bluetooth or RFID. Collect relevant information such as the equipment's user manual, maintenance records, fault history and fault handling solutions. Digitize the above information and integrate it into a database or management system for easy query and use. Use image recognition and processing software to analyze equipment images to detect potential problems or degraded operating conditions. Generate specific operation and maintenance, maintenance guidance reports and inspection work tasks based on image analysis results and equipment history information. The report should include maintenance recommendations, parts that need to be replaced, preventive measures, etc. Feedback the generated report to the maintenance team or relevant personnel. Continuously optimize maintenance guidance information and processes based on actual maintenance results.

[0020] The above steps can achieve the following technical effects: By automatically collecting and analyzing equipment information, maintenance needs can be quickly and accurately determined, shortening equipment maintenance time. The digital and automated process reduces errors and omissions in human judgment. More precise maintenance reduces unnecessary maintenance work and parts replacement, reducing maintenance costs. Regular and accurate maintenance ensures stable operation of equipment and reduces failure rates. Data collected over a long period of time provides data support for future maintenance decisions and helps to continuously improve maintenance strategies. By analyzing the performance of equipment in different locations and environments, more targeted guidance can be provided for the use and maintenance of equipment in specific environments. In summary: the intelligence and automation of equipment maintenance can be effectively improved, costs can be reduced, and efficiency and reliability can be improved.

[0021] Step S2: constructing a three-dimensional virtual testing device based on the image of the device to be tested.

[0022] Specifically, in the process of realizing "constructing a three-dimensional virtual test device based on the image of the device to be tested", it is first necessary to shoot images of the device to be tested from multiple angles to obtain sufficient visual data. These image data provide various perspectives of the device, so that the complete appearance and shape of the device can be captured. Then, these multi-perspective images are processed using SLAM technology (simultaneous localization and mapping) to generate point cloud data. Point cloud data contains the three-dimensional coordinates of the device surface and can accurately reflect the geometric shape of the device. Based on these point cloud data, a three-dimensional spatial model of the device is constructed using three-dimensional modeling technology, and the model is preprocessed, such as denoising and smoothing, to ensure its accuracy and integrity. The final three-dimensional virtual test device can be used for fault detection, analysis and maintenance planning. For example, in the detection of an industrial robot arm, its image is first obtained through multi-perspective shooting, then point cloud data is generated through SLAM technology, and then a three-dimensional model is constructed, and finally the model is optimized, so that maintenance personnel can perform fault analysis and maintenance exercises on the robot in a virtual environment.

[0023] Step S3, obtaining a display position in a virtual interface based on the position of the camera and the position of the device to be detected.

[0024] Specifically, first establish or identify the specific position and orientation of the camera. The specific position and orientation are then achieved by using CV image recognition technology. Similarly, the position of the device to be detected is determined by a preset QR code, RFID tag, or manually input device position coordinates. Use the camera to capture the surrounding environment in real time and identify the spatial layout and feature points through computer vision technology, and use the edge computing optimization algorithm to perform edge optimization on the identified spatial layout and feature points to obtain a more accurate spatial layout and feature points. To correspond and calibrate the camera coordinate system with the coordinate system of the device to be detected, it is necessary to dynamically adjust the conversion parameters between the world coordinate system, camera coordinate system, and device coordinate system through a preset context-aware coordinate conversion scheme.

[0025] Through computer algorithms, the relative position of the device within the camera's field of view is obtained based on the camera's position, orientation, and the device's position.

[0026] The specific display position of the device information in the virtual interface is determined according to the relative position and orientation of the camera and the device. Allow users to interact with the interface by touching the screen, voice commands, head or hand movements, etc., to facilitate querying or obtaining more information. When the position and angle of the camera change, the display position and content in the virtual interface need to be updated in real time to ensure alignment with the real world picture.

[0027] The above steps can achieve the following technical effects: users can intuitively understand the status and location of the equipment through the virtual interface, which improves the comprehensibility of the information. The location and related information of the equipment are accurately indicated in the visual image, reducing the possibility of operational errors. Augmented reality technology can guide users to find the exact location of the maintenance equipment through a virtual path. Users can instantly obtain detailed information and maintenance instructions for the equipment without tedious manual search. Users can interact with the interface, such as zooming in on equipment details, viewing maintenance records, defect handling records, etc., which improves the efficiency of maintenance work. Provide real-time maintenance guidance and support, reduce dependence on paper documents, and improve the convenience of the working environment. In summary, multi-level information display can be achieved, enhancing users' scene understanding ability and interactive experience, thereby improving the efficiency and quality of equipment maintenance and management.

[0028] Step S4, superimposing the maintenance guidance information of the device to be inspected and the three-dimensional virtual inspection device on a display position in the virtual interface to generate a maintenance guidance image centered on the device to be inspected.

[0029] Specifically, the maintenance guidance information of the equipment to be inspected and the three-dimensional virtual test equipment to be inspected are superimposed and displayed at the display position in the virtual interface to generate a maintenance guidance image centered on the equipment to be inspected, which involves the integration of multiple technical steps and components. First, it is necessary to obtain detailed information about the three-dimensional virtual test equipment, including the size, shape, component composition and function of the equipment, which is usually obtained from the technical documents provided by the equipment manufacturer or the three-dimensional model of the equipment. These data provide the basis for subsequent superimposed display. After obtaining the information of the three-dimensional virtual test equipment, it is also necessary to obtain the maintenance guidance information of the equipment. This guidance information includes the steps for maintaining and repairing the equipment, the required tools and methods, etc., which can be obtained through the maintenance manual or technical support documents of the equipment. Then, a real-time image of the equipment to be inspected is obtained through a camera, scanner or other image acquisition tool, or a pre-prepared three-dimensional model of the equipment is used. The image obtained by the image acquisition tool will serve as the basic image of the virtual interface. Subsequently, the image processing software analyzes the acquired equipment image, identifies the various components and marking points on the equipment, and thus matches the structure and component information of the equipment with the image. In this way, the actual appearance of the equipment can be matched with its virtual model to ensure that the virtual image accurately reflects the status of the actual equipment. Next, the maintenance guidance information is superimposed on the equipment image in the form of images, text or icons. This step is usually achieved using augmented reality (AR) technology. AR technology allows maintenance information to be overlaid as layers or virtual objects while displaying the equipment. Users can see this information superimposed on the equipment image through display devices (such as head-mounted displays, smartphones or tablets), thereby providing real-time maintenance guidance. For example, when a user views a mechanical device through an AR headset, the system aligns the virtual 3D model of the device with the image of the actual device. Then, information such as maintenance steps, tool recommendations and component locations are superimposed on the equipment image in the form of transparent layers through AR technology. Users can select different maintenance steps or detailed information by touching the screen, clicking the mouse or voice commands to obtain intuitive and immediate maintenance guidance. This method not only improves the efficiency of maintenance work, but also reduces the risk of operating errors, helping maintenance personnel complete equipment maintenance tasks more accurately.

[0030] In a specific implementation, the step of obtaining information of the device to be detected includes: A working QR code is set on the device to be detected, and information task extraction is performed on the working QR code to obtain extraction task information; The extracted task information is input into the AR glasses to obtain a corresponding detection work task; wherein the detection work task includes an operation process for maintaining the equipment to be detected.

[0031] Specifically, first, a working QR code is set on the device. This QR code contains information related to equipment maintenance and inspection. Next, information is extracted from this working QR code. This means using a device (such as a scanner or camera) to read the data in the QR code and parse the data into usable information. Through information extraction, task information related to the equipment is obtained. This information may include equipment maintenance steps, inspection standards, operation instructions, etc. Then, the extracted task information is input into AR glasses. AR glasses are a wearable device that can overlay virtual information in the user's field of view. After receiving the task information, AR glasses can convert this information into specific inspection tasks. These tasks will be displayed in the user's field of view in the form of virtual prompts to guide users on how to inspect and maintain the equipment. Finally, these inspection tasks include detailed maintenance work processes. Users can see step instructions, precautions, operation methods, etc. through AR glasses, so as to complete the inspection and maintenance of the equipment more effectively. In summary, this step describes a process of assisting equipment maintenance and inspection through QR codes and AR technology. The QR code provides task information, and AR glasses present this information to users in an intuitive way to help them complete related inspection and maintenance tasks.

[0032] In a specific implementation, obtaining the display position in the virtual interface based on the position of the camera and the position of the device to be detected includes: Obtaining the position of the three-dimensional virtual testing device based on the position of the device to be tested; Calculating the relative position between the camera and the three-dimensional virtual measuring device to obtain an initial relative distance value between the two; The virtual interface is adjusted based on the initial relative distance value to obtain a display position in the adjusted virtual interface.

[0033] Specifically, it is first necessary to determine the display position in the virtual interface based on the position of the camera and the device to be detected. The first step in this process is to accurately obtain the position of the device to be detected in the virtual space. This usually relies on a three-dimensional virtual model of the device, which includes the size, shape, position of each component and other spatial information of the device. This information can be obtained through a preset three-dimensional model of the device or three-dimensional data obtained through real-time scanning. Once the position of the device to be detected in the virtual space is determined, the relative position between the camera and the device to be detected needs to be calculated. This calculation process involves obtaining the actual position of the camera and the specific position of the device to be detected in the physical world. Usually, the position of the camera can be determined by its built-in positioning system or external positioning marks. The position of the device to be detected is obtained by its coordinates in the virtual space. After obtaining the relative position between the camera and the device to be detected, the system calculates the initial relative distance value between the two. This step is achieved through geometric calculations, which include measuring the straight-line distance between the camera and the device, or calculating the spatial distance between them in three-dimensional space.

[0034] The initial relative distance value provides basic data for the subsequent adjustment of the virtual interface, so that the display position of the virtual interface can be accurately adjusted according to the actual distance. Based on the calculated initial relative distance value, the system will adjust the virtual interface to ensure the accuracy of the display position. This means that the display position on the virtual interface needs to be adjusted accordingly according to the actual relative distance between the camera and the device to be detected. Specifically, the system uses a piecewise function method to achieve this adjustment. This method uses different scaling strategies according to different distance ranges: when the distance is less than the preset close distance threshold, the system will use a larger scaling factor to make the device on the virtual interface larger and clearer; when the distance is between the close distance threshold and the medium distance threshold, the system will use a medium scaling factor to maintain a suitable display size; when the distance exceeds the medium distance threshold, the system uses a smaller scaling factor to ensure that the entire device is still visible. This piecewise function method can better adapt to the observation needs at different distances and provide a more natural and intuitive visual experience. For example, if the camera is close to the device to be detected, the device display position on the virtual interface needs to be enlarged or moved closer to match the actual observation effect. Conversely, if the distance is far, it needs to be reduced or moved away. The display position in the adjusted virtual interface should ensure that the display effect of the virtual device is consistent with the look and feel of the actual device in the physical space. In this way, the device position and size seen by the user in the virtual interface are consistent with the observation of the actual device, which can provide more accurate inspection and maintenance guidance. This adjustment not only improves the device visualization effect in virtual reality, but also enhances the interactive experience between the user and the device, so that the virtual interface can better reflect the actual situation, thereby improving work efficiency and accuracy.

[0035] In a specific implementation, the maintenance guidance information of the device to be inspected and the three-dimensional virtual device to be inspected are superimposed and displayed at a display position in the virtual interface, and the maintenance guidance information is generated with the device to be inspected as the center, including: Inputting the maintenance guidance information of the device to be tested and the three-dimensional virtual test device to be tested into a preset computer vision model for fusion to obtain a fused image; Obtaining, for the three-dimensional virtual testing device to be tested, an operation and maintenance period of the three-dimensional virtual testing device to be tested; Performing task analysis on the three-dimensional virtual testing equipment to be tested based on the operation and maintenance cycle to formulate a testing task, and obtaining an AR task and an AR task item based on the testing task; Based on the AR operation task and the AR operation task item, the position of the fused image is adjusted through a preset dynamic line of sight tracking adjustment method to obtain a target image; wherein the target image is a maintenance guidance image displayed with the equipment to be repaired as the center.

[0036] Specifically, when the maintenance guidance information of the equipment to be tested and the three-dimensional virtual test equipment to be tested are superimposed and displayed on the virtual interface, the maintenance guidance information of the equipment to be tested and the three-dimensional virtual test equipment need to be fused first. This step involves integrating the maintenance guidance information (such as equipment fault diagnosis steps, maintenance manuals, instructions for replacing parts, etc.) with the image data of the three-dimensional virtual test equipment. To this end, the two parts of information need to be input into a preset computer vision model, which uses a semantic segmentation algorithm, such as a U-Net or SegNet deep learning network, to segment the three-dimensional virtual test equipment model and identify each key component and area. The model matches the maintenance guidance information with the identified components, and accurately maps the corresponding maintenance guidance information to the corresponding components of the three-dimensional virtual test equipment model through image processing and data fusion algorithms, thereby generating a fused image. For example, for a part that needs to be replaced, the model will accurately superimpose the replacement instructions on the position of the part in the three-dimensional model. This semantic segmentation-based method can achieve accurate correspondence between maintenance information and equipment models, thereby generating a three-dimensional virtual test equipment image that integrates maintenance guidance information. Next, in order to ensure that the fused image can effectively reflect the operation and maintenance cycle of the equipment, it is necessary to obtain the operation and maintenance cycle of the 3D virtual test equipment to be inspected. The operation and maintenance cycle information usually includes the equipment's maintenance schedule, historical fault records, and regular inspection requirements. This information can help analyze the current status and maintenance needs of the equipment, and then formulate appropriate inspection tasks. After obtaining the operation and maintenance cycle information, it is necessary to perform task analysis on the 3D virtual test equipment to be inspected based on this information. Task analysis includes determining the inspection and maintenance tasks that need to be performed, such as inspecting specific parts of the equipment, replacing damaged parts, etc. Based on the results of the task analysis, the system will generate AR (augmented reality) work tasks and their specific items, which will guide maintenance personnel to perform operations and ensure that all necessary steps are followed. Then, according to the AR work tasks and task items, the fused image is adjusted in position through the preset dynamic line of sight tracking adjustment method.

[0037] The specific adjustment process of the preset dynamic sight tracking adjustment method includes calculating the position of the current gaze point in the AR interface to identify the area of ​​concern, matching the identified area of ​​concern with the AR task item to determine the task item that needs to be adjusted, calculating a new optimal display position for the task item that needs to be adjusted, applying the calculated optimal display position to the fused image based on the current sight direction and the current image layout, and finally smoothly transitioning the task item to the new position to avoid visual interference. Repeat the above steps in real time to dynamically adjust the image according to the changes in sight.

[0038] This adjustment process ensures that the maintenance guidance information is aligned with the 3D virtual device image, so that the guidance information is accurately displayed at the relevant position of the device. By accurately positioning and adjusting the fused image, the generated target image will be centered on the device to be repaired, intuitively displaying the maintenance guidance information, and helping maintenance personnel to clearly view and understand the maintenance steps on the virtual interface. In order to make the solution better understood, an example can be given. For example, suppose we need to maintain an industrial pump for a device to be tested. In this process, first, the system will fuse the pump's maintenance guidance information (such as maintenance steps, necessary tools, and component replacement instructions) with the 3D virtual model of the pump. Then, the system will obtain the pump's operation and maintenance cycle information, perform task analysis, and formulate specific AR work tasks, such as checking the pump's seals and replacing filters. Based on these tasks, the system will adjust the position in the fused image so that the maintenance guidance information is accurately covered on the corresponding position of the virtual model of the pump, thereby generating a target image centered on the pump. In the virtual interface, maintenance personnel can perform precise maintenance operations based on the guidance information on the target image to ensure the efficiency and accuracy of the maintenance process. This method not only improves the accuracy and efficiency of equipment maintenance, but also reduces the possibility of human error, allowing maintenance personnel to obtain detailed operational guidance in a virtual environment, thereby improving the overall maintenance quality and equipment operational reliability.

[0039] In a specific implementation, it also includes: obtaining operation information of maintenance personnel; wherein the operation information is information that maintenance personnel need to follow specific steps to perform operations when performing equipment inspection and maintenance; Extracting the operation information to obtain feedback information from the operator; Input the feedback information into a preset data processing model for data processing and analysis to obtain a maintenance form for the equipment to be tested; Encrypting the maintenance form of the device to be detected to obtain an encrypted maintenance table, and storing the encrypted maintenance table; Alternatively, after the step of obtaining the operation information of the maintenance personnel, the following steps are included: Extracting operation information from the operation information to obtain operation feature information; The operation characteristic information is input into a preset assessment analysis model for analysis or the target personnel remotely view the operation characteristic information of the operation operation background to obtain the analysis result of the assessment; wherein the assessment analysis model has information of the pre-input judgment rules; Obtaining an assessment report based on the analysis results of the assessment; The operation performance is calculated based on the evaluation report to obtain the operation performance of the maintenance personnel.

[0040] Specifically, the operation information of the maintenance personnel needs to be obtained first. This operation information refers to the detailed record of the maintenance personnel's operation according to specific steps during the equipment inspection and maintenance process. This information usually includes the operation steps, execution time, operation sequence, tools and equipment used, etc. This information can be collected by various means, such as sensors on the equipment, maintenance record systems, or manual input. Once the operation information of the maintenance personnel is obtained, the next step is to extract this information to obtain the feedback information of the operator. Information extraction involves extracting key feedback information from the collected operation data, such as whether the operation is completed according to the prescribed steps, whether there are problems or abnormalities, etc. This feedback information is crucial for subsequent data processing and analysis. Next, the extracted feedback information is input into the preset data processing model for data processing and analysis to generate a maintenance form for the equipment to be tested. The data processing model is usually a trained algorithm system that can analyze the feedback information, identify problems or deficiencies in the equipment maintenance process, and generate a detailed maintenance form. This form records the maintenance status of the equipment, the inspection content, the existing problems, and the matters that need further processing. The generated maintenance form needs to be encrypted to ensure its security and privacy. The encrypted maintenance form, i.e. the encrypted maintenance sheet, will be stored in a secure database for future reference and review. After obtaining the maintenance personnel's operation information, additional analysis steps can be performed. This includes extracting operation feature information from the operation information. Operation feature information extraction is to extract representative features from the operation records, such as the accuracy and efficiency of the operation, whether the predetermined operation standards are followed, etc. These feature information will be input into the preset evaluation analysis model for analysis. The evaluation analysis model is a system that scores operations according to pre-set evaluation rules. It can generate evaluation results based on the maintenance personnel's operation feature information. Among them, the pre-set evaluation rules include: Time efficiency: Evaluates the time it takes maintenance personnel to complete a task compared to the standard time, calculated as (standard time - actual completion time) / standard time * 100.

[0041] Operational accuracy: measures the ratio of correctly completed steps to the total number of steps, calculated by (correct steps / total steps) * 100.

[0042] Safety Compliance: All safety regulations are checked for compliance, and 10 points will be deducted for each violation of safety regulations.

[0043] Tool use: Assess the correct selection and use of tools, calculated as (number of correct tool uses / total number of tool uses) * 100.

[0044] Problem diagnosis: Assess the ability to correctly diagnose problems, evaluated by (number of correct diagnoses / total number of diagnoses) * 100.

[0045] For example, if the maintenance personnel complete the operation according to the standard steps when the equipment to be inspected is a machine, the system will obtain and extract the operation information to generate the operator's feedback information. Then, the feedback information will be input into the data processing model to generate a maintenance form. This maintenance form is encrypted and stored in the system to ensure the security of the information. At the same time, the operation feature information is extracted and input into the evaluation analysis model for evaluation. The model calculates the evaluation results according to the set evaluation rules. Finally, the evaluation results are used to generate an evaluation report, and the maintenance personnel's operation performance is calculated based on this report. This method can not only record the operation process of the maintenance personnel in detail, but also evaluate and assess the operation quality to ensure the standardization and efficiency of the maintenance work. At the same time, the encryption and storage mechanism ensures the security of the maintenance records and data privacy. In practical applications, this system can significantly improve the management level of maintenance work and provide scientific and accurate support for equipment maintenance.

[0046] In a specific implementation, it also includes: when it is found that the device to be inspected has a defect, the device to be inspected generates a defect document; Acquire defect problem information inquired by maintenance personnel based on the defect document; Input the maintenance personnel's inquiry question information into a preset maintenance information knowledge base for information matching or remotely inquire the target personnel to obtain the answer corresponding to the maintenance personnel's inquiry question information; Analyze the maintenance personnel's inquiry information to obtain answers related to the maintenance personnel's inquiry information; Mining the inquiry question information of the maintenance personnel to obtain deeper question information of the inquiry information of the maintenance personnel; The deeper question information of the maintenance personnel's inquiry information and the answers associated with the inquiry question information are fused to obtain fused information, and the fused information is added to the inspection and maintenance information knowledge base.

[0047] Specifically, first, a defect document needs to be generated when a defect is found in the equipment to be inspected. The document records the specific information of the equipment defect in detail, such as the nature, location and possible impact of the defect. Generating a defect document is the starting point of the entire problem handling process, and it provides basic data for subsequent analysis and processing. Next, based on the generated defect document, the system needs to obtain the defect problem information asked by the maintenance personnel. These problem information are usually specific questions raised by the maintenance personnel when handling defects, such as the cause of the defect, repair steps, necessary tools, etc. These questions can be obtained through the maintenance personnel's inquiry records, system input or direct communication. After obtaining the maintenance personnel's inquiry question information, the system enters this information into the preset maintenance and maintenance information knowledge base for information matching. The maintenance and maintenance information knowledge base is a large database containing various maintenance and maintenance information, which stores knowledge, experience and solutions about equipment maintenance. Through information matching, the system can find answers related to the questions asked by the maintenance personnel in the knowledge base. If there is no direct answer in the knowledge base, the system can also choose to remotely ask the target person or expert to obtain a more accurate answer. This method ensures that the maintenance personnel can quickly obtain accurate information, thereby improving the efficiency of problem solving. In addition, the system also needs to mine the maintenance personnel's inquiry information to discover deeper problem information. Specifically, use natural language processing technology to perform semantic analysis on the inquiry questions and extract keywords and topics; use the knowledge graph in the field of equipment maintenance to achieve semantic expansion of the questions; apply text clustering algorithms to analyze the extracted keywords and topics to discover potential problem patterns; extract key information such as equipment components and fault types; integrate the discovered problem patterns and extracted key information to obtain the core content, thereby generating deeper problem information.

[0048] This means not only being satisfied with answering direct questions, but also exploring potential and more complex problems. For example, maintenance personnel may ask about the solution to a defect, and through mining, potential problems or other unnoticed factors related to the defect can be discovered. The mining process includes in-depth analysis of the background of the problem, the operation of the equipment, and the maintenance history to provide a more comprehensive solution. Finally, the system fuses the deeper question information of the maintenance personnel's inquiry information with the relevant answers. This fusion process involves integrating information from different sources to generate a more complete and useful answer. The fused information will be supplemented to the maintenance information knowledge base to enrich the content of the knowledge base and improve the knowledge coverage and accuracy of the system. In this way, similar questions can be better answered in the future, thereby improving the efficiency and intelligence level of the entire maintenance system. For example, suppose a piece of equipment has a mechanical failure, and the defect document generated records the specific information of the failure. When handling the failure, the maintenance personnel raised questions about the cause of the failure and the repair steps. The system first matches some relevant answers from the knowledge base. If the information in the knowledge base is insufficient, the system will obtain supplementary answers through remote expert consultation. After getting the answers, the system analyzes whether these answers solve the maintenance personnel's specific problems and further explores potential problems related to the fault. Finally, the system combines this information to generate a comprehensive answer and updates it to the knowledge base for future use. Through this process, the system continuously optimizes and expands its knowledge base and improves its problem-solving capabilities.

[0049] In a specific implementation, the construction of a three-dimensional virtual testing device based on the image of the device to be tested includes: Taking images of the device to be detected to obtain images of the device to be detected from different viewing angles; The point cloud data of the image of the device to be detected at different viewing angles is obtained through the preset SLAM technology; wherein the point cloud data is a data set composed of a large number of three-dimensional point coordinates on the image of the device to be detected at different viewing angles; Based on the point cloud data, construct a three-dimensional spatial model of the device to be detected; Preprocessing the three-dimensional space model of the device to be detected to obtain a target three-dimensional space model of the device to be detected; Adding annotation information to the target three-dimensional space model through a preset annotation tool to obtain a three-dimensional space model with annotations; wherein the annotation information includes the fault location, maintenance steps and parts replacement instructions of the equipment to be detected; The annotated three-dimensional space model is rendered to obtain a rendered three-dimensional space model; wherein the rendered three-dimensional space model is used for visual display.

[0050] Specifically, it is necessary to first build a 3D virtual test device based on the image of the device to be tested. This process includes multiple steps, involving a complete process from image capture to the final rendering of the 3D model. First, capturing images of the device to be tested is the starting point for building a 3D virtual test device. During the shooting process, images of the device need to be obtained from different perspectives to ensure that all angles of the device are fully covered. These images will be used as basic data for subsequent processing to generate a 3D spatial model of the device. In order to obtain these images, a high-resolution camera or other image acquisition device can be used to shoot the device from different angles and positions to ensure that every detail is captured. Next, the point cloud data of the images of the device to be tested from different perspectives is obtained through the preset SLAM (Simultaneous Localization and Mapping) technology. SLAM technology can convert images into point cloud data, which is a data set composed of a large number of 3D point coordinates that can accurately describe the 3D shape and structure of the device. Point cloud data is generated by feature points identified in the image, which reflect the position and form of the device in space. Through point cloud data, a highly accurate 3D representation of the device surface and its details can be obtained. Based on these point cloud data, a 3D spatial model of the device to be tested is constructed. The process of building a 3D spatial model includes converting point cloud data into a 3D mesh model that can be used for further processing. This model accurately reflects the actual geometry of the equipment and provides a basis for subsequent analysis and processing. The construction of the 3D model requires the use of computer graphics and 3D modeling technology to integrate the point cloud data into a continuous 3D surface through algorithms. The constructed 3D spatial model is preprocessed to obtain the target 3D spatial model of the equipment to be tested. This preprocessing process includes removing noise, filling missing parts, smoothing the model surface, etc. to improve the accuracy and quality of the model. The preprocessed 3D spatial model will more accurately reflect the actual status of the equipment and provide a clearer basis for subsequent annotation and rendering. Through the preset annotation tool, annotation information is added to the target 3D spatial model to obtain a 3D spatial model with annotations. The annotation information includes the fault location, maintenance steps, and part replacement instructions of the equipment to be tested. These annotation information are crucial for the maintenance and repair of the equipment. They help maintenance personnel quickly identify the fault location, understand the repair steps, and identify the parts that need to be replaced. The annotation tool can be a dedicated 3D modeling software or a customized annotation system that adds relevant information to the 3D model through an interactive interface. Finally, the annotated 3D space model is rendered to obtain a rendered 3D space model. The rendering process converts the 3D model into a visual image, so that the appearance, fault location and maintenance information of the equipment can be displayed to the user in an intuitive way.The rendered 3D spatial model can use graphics processing technology to adjust the color, lighting and material of the model to enhance the visual effect, so that maintenance personnel can more clearly understand the status and maintenance requirements of the equipment. The rendering results can be used in training, fault analysis and actual maintenance operations to improve the efficiency and accuracy of maintenance work. For example, suppose an industrial machine has a fault. First, images of the machine are taken from multiple angles by a high-resolution camera. Then, SLAM technology is used to generate point cloud data of these images to build a 3D spatial model of the machine. Next, the model is preprocessed to ensure its accuracy and completeness. Using annotation tools, the fault location, maintenance steps and parts replacement instructions of the machine are annotated into the model. Finally, a visual 3D spatial model is generated through rendering technology, so that maintenance personnel can intuitively see the status of the machine and the maintenance guide. This process will significantly improve the efficiency and accuracy of equipment maintenance and ensure that maintenance work can be carried out accurately.

[0051] The above describes the maintenance method based on AR glasses in the embodiment of the present invention. The following describes the maintenance system based on AR glasses in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, an inspection and maintenance system based on AR glasses includes: The acquisition module 21 is used to acquire information of the device to be detected; wherein the information of the device to be detected includes an image of the device to be detected, a location of the device to be detected, and maintenance guidance information and detection work tasks of the device to be detected; A construction module 22, used for constructing a three-dimensional virtual testing device based on the image of the device to be tested; A display module 23, used to obtain a display position in a virtual interface based on the position of the camera and the position of the three-dimensional virtual measuring device; The superposition module 24 is used to superimpose the maintenance guidance information of the device to be inspected and the three-dimensional virtual test device on a display position in the virtual interface to generate a maintenance guidance image centered on the device to be inspected.

[0052] In this embodiment, for the specific implementation of each unit in the above system embodiment, please refer to the above method embodiment, which will not be repeated here.

[0053] Reference Figure 3 The present invention also provides a computer device in an embodiment, wherein the internal structure of the computer device can be as follows: Figure 3As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0054] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0055] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0056] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0057] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0058] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A maintenance method based on AR glasses, characterized in that: The AR glasses include a virtual interface and a camera, and the inspection and maintenance method includes: Acquire information of the device to be detected; wherein the information of the device to be detected includes an image of the device to be detected, a location of the device to be detected, and maintenance guidance information and detection work tasks of the device to be detected; Constructing a three-dimensional virtual testing device based on the image of the device to be tested; Based on the position of the camera and the position of the device to be detected, obtaining a display position in a virtual interface; The maintenance guidance information of the device to be inspected and the three-dimensional virtual inspection device are superimposed and displayed at a display position in the virtual interface to generate a maintenance guidance image centered on the device to be inspected.

2. The AR glasses-based maintenance method according to claim 1, characterized in that: The step of obtaining information of the device to be detected includes: A working QR code is set on the device to be detected, and information task extraction is performed on the working QR code to obtain extraction task information; The extraction task information is input into the AR glasses to obtain the detection work task; wherein the detection work task includes an operation process for maintaining the equipment to be detected.

3. The AR glasses-based maintenance method according to claim 1, characterized in that: The obtaining a display position in a virtual interface based on the position of the camera and the position of the device to be detected includes: Obtaining the position of the three-dimensional virtual testing device based on the position of the device to be tested; Calculating the relative position between the camera and the three-dimensional virtual measuring device to obtain an initial relative distance value between the two; Based on the initial relative distance value, a display position for displaying the image is determined in the virtual interface.

4. The AR glasses-based maintenance method according to claim 1, characterized in that: The method of superimposing the maintenance guidance information of the device to be inspected and the three-dimensional virtual inspection device on a display position in a virtual interface to generate a maintenance guidance image centered on the device to be inspected includes: Inputting the maintenance guidance information of the equipment to be inspected and the three-dimensional virtual inspection equipment into a preset computer vision model for fusion to obtain a fused image; Obtaining, for the three-dimensional virtual testing device to be tested, an operation and maintenance period of the three-dimensional virtual testing device to be tested; Performing task analysis on the three-dimensional virtual testing equipment to be tested based on the operation and maintenance cycle to obtain the testing work task, and obtaining the AR work task and AR work task item based on the testing work task; Based on the AR operation task and AR operation task item, the position of the fused image is adjusted by a preset dynamic line of sight tracking adjustment method to obtain a target image; wherein the target image is a maintenance guidance image displayed with the equipment to be repaired as the center.

5. The AR glasses-based maintenance method according to claim 1, characterized in that: Also includes: Acquire operation information of maintenance personnel; wherein the operation information is information that maintenance personnel need to follow specific steps to perform operations when performing equipment inspection and maintenance; Extracting the operation information to obtain feedback information from the operator; Input the feedback information into a preset data processing model for data processing and analysis to obtain a maintenance form for the equipment to be tested; Encrypting the maintenance form of the device to be detected to obtain an encrypted maintenance table, and storing the encrypted maintenance table; Alternatively, after the step of obtaining the operation information of the maintenance personnel, the method further comprises: Extracting operation information from the operation information to obtain operation feature information; The operation characteristic information is input into a preset assessment analysis model for analysis or the target personnel remotely view the operation characteristic information of the operation operation background to obtain the analysis result of the assessment; wherein the assessment analysis model includes the pre-input evaluation rules; Obtaining an assessment report based on the analysis results of the assessment; An operation performance calculation is performed based on the evaluation report to obtain the operation performance of the maintenance personnel.

6. The AR glasses-based maintenance method according to claim 1, characterized in that: Also includes: When the equipment to be inspected is found to have defects, a defect document is generated for the equipment to be inspected; Acquire defect problem information inquired by maintenance personnel based on the defect document; Input the defect problem information of the maintenance personnel into a preset maintenance information knowledge base for information matching or remotely inquire the target personnel to obtain the answer corresponding to the inquiry question information of the maintenance personnel; Analyze the maintenance personnel's inquiry information to obtain answers related to the maintenance personnel's inquiry information; Processing the inquiry information of the maintenance personnel to obtain deeper question information of the inquiry information of the maintenance personnel; The deeper question information of the maintenance personnel's inquiry information and the answers associated with the inquiry question information are fused to obtain fused information, and the fused information is added to the inspection and maintenance information knowledge base.

7. The AR glasses-based maintenance method according to claim 1, characterized in that: The method of constructing a three-dimensional virtual testing device based on the image of the device to be tested includes: Taking images of the device to be detected to obtain images of the device to be detected from different viewing angles; The point cloud data of the image of the device to be detected at different viewing angles is obtained by using a preset SLAM technology; wherein the point cloud data is a data set composed of three-dimensional point coordinates on the image of the device to be detected at different viewing angles; Based on the point cloud data, construct a three-dimensional spatial model of the device to be detected; Preprocessing the three-dimensional space model of the device to be detected to obtain a target three-dimensional space model of the device to be detected; Adding annotation information to the target three-dimensional space model through a preset annotation tool to obtain a three-dimensional space model with annotations; wherein the annotation information includes the fault location, maintenance steps and parts replacement instructions of the equipment to be detected; Rendering the annotated three-dimensional space model to obtain a rendered three-dimensional space model, and using the rendered three-dimensional space model as a three-dimensional virtual testing device to be tested; wherein the rendered three-dimensional space model is used for visual display; The three-dimensional virtual testing device to be tested is stored in a device library.

8. An inspection and maintenance system based on AR glasses, characterized in that: The AR glasses include a virtual interface and a camera, and the inspection and maintenance system includes: An acquisition module is used to acquire information of the device to be detected; wherein the information of the device to be detected includes an image of the device to be detected, a location of the device to be detected, and maintenance guidance information and detection work tasks of the device to be detected; A construction module, used for constructing a three-dimensional virtual testing device based on the image of the device to be tested; A display module, used for obtaining a display position in a virtual interface based on the position of the camera and the position of the three-dimensional virtual measuring device; The superposition module is used to superimpose the maintenance guidance information of the device to be inspected and the three-dimensional virtual test device on a display position in the virtual interface to generate a maintenance guidance image centered on the device to be inspected.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images