Mechanical equipment AR real-time tracking maintenance method based on YOLOv11 and coordinate transformation

Abstract

The invention discloses a mechanical equipment AR (Augmented Reality) real-time tracking maintenance method based on YOLOv11 and coordinate transformation. The method comprises the following steps of collecting equipment operation data through a sensor and evaluating a health state; acquiring an image through an AR equipment camera and transmitting the image to a server; the server calls a YOLOv11 model to carry out equipment identification, and outputs an equipment category and a pixel coordinate; establishing a transformation matrix through a calibration process, and converting the image coordinates into AR equipment display coordinates; and finally, fusing the state information with the display coordinates, and carrying out visual projection in a virtual-real superposition form through AR (Augmented Reality) equipment. The invention further discloses a real-time tracking maintenance system which comprises a state monitoring module, an equipment identification module and a data visualization module. Through fusion of YOLOv11 high-precision target detection, coordinate transformation and multi-source data interaction, the problems that an existing AR maintenance system is poor in equipment recognition robustness, inaccurate in space registration and disjointed with underlying monitoring are solved, and real-time, accurate and visual intelligent maintenance of the equipment state is achieved.

Description

Technical Field

[0001] This invention relates to the field of mechanical equipment condition monitoring and intelligent maintenance technology, and in particular to a real-time AR tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation. Background Technology

[0002] In the context of Industry 4.0, the demand for intelligent and real-time maintenance of machinery and equipment is becoming increasingly urgent. Traditional maintenance methods rely heavily on reactive repairs and human experience, making it difficult to predict potential faults and easily leading to unplanned downtime and production losses. Existing condition monitoring systems are usually tied to fixed terminals, forcing operators to frequently switch between on-site equipment and screens, resulting in high cognitive load and information delays.

[0003] While augmented reality technology has been introduced into industrial maintenance, existing solutions still have significant shortcomings: some systems rely on preset markers for tracking, resulting in poor adaptability in complex scenarios; the equipment recognition process is susceptible to interference from lighting and occlusion, leading to insufficient real-time performance and robustness; the spatial registration accuracy between virtual information and real equipment is insufficient, resulting in cluttered visualization effects and inaccurate positioning; and most systems have failed to deeply integrate with underlying condition monitoring and predictive maintenance algorithms, remaining at the level of passive maintenance guidance and failing to form a closed-loop intelligent maintenance system.

[0004] Furthermore, while advanced target detection algorithms such as YOLOv11 have improved detection performance, they have not yet been effectively integrated with AR technology and industrial sensor data to form a comprehensive intelligent maintenance solution. Therefore, it is necessary to construct an intelligent maintenance solution that integrates high-precision identification, real-time diagnosis, and accurate visualization. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time AR tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation. It integrates deep learning target detection, homogeneous coordinate transformation and multi-source sensor data analysis to achieve accurate identification, stable registration and visual status presentation of mechanical equipment in augmented reality environment. It solves the technical problems of poor equipment identification robustness, inaccurate spatial position registration and disconnect between AR display and underlying monitoring data in existing AR maintenance systems.

[0006] To achieve the above objectives, the specific technical solution of the AR real-time tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation of the present invention is as follows: A real-time AR tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation includes the following steps: S1: Status monitoring, which collects operating data of mechanical equipment through industrial sensors. The industrial sensors are deployed in key parts of the mechanical equipment such as gearboxes, bearing housings, and reducers. The collected operating data mainly includes acceleration and current data. The operating data is automatically processed to evaluate the health status of the equipment, and the processed equipment status indicators are stored in the database. S2: Device identification, used to collect image data of mechanical equipment through the camera of AR wearable device, and transmit the image data to the server; S3: The image data is processed by the YOLOv11 target detection model deployed on the server to achieve accurate detection of the target device and output the device pixel coordinates; S4: Coordinate Transformation, used to establish a transformation matrix between the device image and the AR wearable device display unit through homogeneous coordinate transformation method, converting the device pixel coordinates output by the YOLOv11 model into the pixel coordinates of the AR device display unit, so as to realize real-time tracking and visualization of the device position; S5: Data visualization is used to call device status indicators from the database through the server, associate the device status indicators with the device identification results, and project the device status information and maintenance guidance to the user's field of vision through the display unit of the AR wearable device to realize the real-time presentation of device status information.

[0007] Furthermore, the status monitoring module is integrated with the predictive maintenance system (PMS) and transmits the collected operating data to the server via industrial Ethernet. The server analyzes the equipment operating status through preset fault diagnosis algorithms and anomaly detection algorithms. The equipment operating status includes health status, early warning status, and alarm status.

[0008] Furthermore, the device identification module transmits image data to the server via the Real-time Message Transfer Protocol (RTMP). The parameters of the image data are configured as follows: resolution 640×480 pixels, frame rate 30fps, H.264 encoding format, and bit rate 1200kbps. The server deploys a YOLOv11 target detection model trained on an industrial equipment dataset to process the image data in real time. Through multi-scale feature fusion and spatial attention mechanism, it achieves accurate detection of target devices and outputs device category, confidence level, and device pixel coordinates.

[0009] Furthermore, the coordinate transformation matrix is ​​established in the following manner, specifically including the following steps: Step 1: Design a calibration board with a rectangular pattern, detect the rectangular pattern of the calibration board using the trained YOLOv11 model, and record its pixel coordinates in the image; Step 2: Preset a rectangular frame on the display unit of the AR wearable device, manually adjust the preset rectangular frame to align with the rectangular pattern on the calibration board, and record the pixel coordinates of the preset rectangular frame; Step 3: Based on the pixel coordinates of the rectangular pattern on the calibration board and the pixel coordinates of the preset rectangular frame, calculate the coordinate transformation matrix through translation and scaling transformations. The coordinate transformation matrix The expression is: YOLOv11 output pixel P img : P img =[xcyc1] H The corresponding AR display coordinates are: P ar =HP img in, , They are respectively shaft and Scaling factor for the direction of the axis. , They are respectively shaft and The translation distance along the axis is used to substitute the device pixel coordinates output by the YOLOv11 model into the transformation matrix, which is then converted into the pixel coordinates of the AR device display unit, thereby realizing real-time tracking and visualization of the device position.

[0010] Furthermore, the server calls the device status indicators in the database through the Simple Object Access Protocol (SOAP); and serializes the device status indicators and device location data through the WebSocket protocol and transmits them to the AR wearable device to ensure the real-time performance and stability of data transmission.

[0011] Furthermore, the equipment status information presented by the data visualization module includes indicators such as equipment name, number, data acquisition time, alarm level, maintenance instructions, sampling frequency, and vibration signal; the time-domain indicators of the vibration signal include root mean square value, kurtosis factor, skewness factor, peak value, waveform factor, impulse factor, average amplitude, and root mean square amplitude.

[0012] Furthermore, the vibration signal time-domain index used by the data visualization module includes the root mean square (RMS) value, which is expressed as follows: Peak value: Kurtosis: Skewness: Crest Factor: Where x i Let N be the i-th vibration acceleration sampling point, and N be the number of sampling points. X is the mean.

[0013] Furthermore, the state monitoring module performs a Fast Fourier Transform (FFT) on the vibration signal: Where N is the number of sampling points, and the peak frequency f is extracted from the spectrum. peak As an indicator for equipment health diagnosis.

[0014] Furthermore, the latency D of the AR real-time tracking system includes image acquisition latency Dcap, network transmission latency Dnet, and model inference latency Dinfer, which is calculated as follows: And it meets the real-time requirement: D≤50 ms.

[0015] This invention also discloses specific modules required for implementing a real-time AR tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation, including: The condition monitoring module consists of an accelerometer, a current sensor, and a data preprocessing unit. The sensors are deployed in key parts of the mechanical equipment, such as gearboxes and bearing housings, to collect signals such as vibration and current during equipment operation. The preprocessing unit transmits the collected raw signals to the server via industrial Ethernet. The server uses the fault diagnosis and anomaly detection algorithms preset by the predictive maintenance system to assess the health status of the equipment (health / early warning / alarm) and stores the processed status indicators in the database.

[0016] Device recognition module: includes a wearable AR device equipped with an autofocus camera and a server deploying a YOLOv11 target detection model; the AR device is used to collect image data of the target mechanical equipment (640×480 pixels resolution, 30fps frame rate) and transmit it to the server via the RTMP protocol; the server processes the image data in real time using the YOLOv11 model, and outputs the device category, detection confidence score and pixel coordinates in the image coordinate system to achieve accurate identification and positioning of the target device.

[0017] The data visualization module includes a coordinate transformation unit and an AR device display unit. The coordinate transformation unit establishes a mapping relationship between image coordinates and AR device coordinates based on a customized rectangular calibration plate. It converts the pixel coordinates output by the device recognition module into AR device display coordinates through a transformation matrix. The AR display unit receives the device status indicators and the converted coordinates transmitted from the server and projects the device name, health status, vibration parameters, and maintenance instructions to the user's field of vision in a virtual-real overlay format, realizing an intuitive presentation and real-time tracking of the device status.

[0018] The AR real-time tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation, as proposed in this invention, has the following advantages: (1) High equipment recognition accuracy: The present invention uses the YOLOv11 target detection model to quickly extract features and locate targets in mechanical equipment appearance images. Compared with traditional SIFT, SURF and other algorithms, it has higher robustness in complex industrial scenarios and can achieve stable recognition of equipment of different shapes.

[0019] (2) Accurate and stable spatial positioning: By establishing a homogeneous coordinate transformation matrix, the image coordinates and AR device coordinates are unified into the same spatial reference system, realizing real-time updates of the target device position and effectively avoiding the problem of virtual and real misalignment caused by slight movement of AR device.

[0020] (3) Accurate and consistent AR maintenance information projection: The spatial position of the equipment obtained by coordinate transformation is used to accurately drive the AR overlay layer, so that the health status, maintenance prompts, model annotations and other information are accurately attached to the surface of the equipment, improving the readability and consistency of AR-assisted maintenance.

[0021] (4) Support for real-time tracking under complex working conditions: The method described in this invention can maintain stable tracking capability in environments such as equipment vibration, light changes, and partial occlusion, and is suitable for industrial scenarios such as high noise, high dynamics, and multiple reflections of mechanical equipment.

[0022] (5) Visualize the status of equipment: After the data from industrial sensors is processed by the server, it is synchronously overlaid on the AR display interface, so that users can obtain the real-time operating status, alarm information and health indicators of the equipment without switching terminals, thus improving maintenance efficiency. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a real-time AR tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation, according to the present invention. Figure 2 This is a schematic diagram of the system structure of the present invention; Figure 3 This invention uses a calibration board to assist in establishing a coordinate mapping relationship between the device image and the display screen of the AR wearable device; Figure 4 This is a diagram showing the two-dimensional coordinate transformation mapping relationship of the present invention; Figure 5 This is a diagram illustrating the system application effect of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0026] The following is a reference to the appendix. Figure 1 To be continued Figure 5 This invention describes a real-time AR tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation, with reference to the appendix. Figure 1 and attached Figure 2 As shown, the specific implementation steps are as follows: S1. Construct a condition monitoring module by deploying industrial-grade sensors in key operating parts of the mechanical equipment (including but not limited to bearing housings, gearboxes, motor housings, and couplings). These sensors include acceleration sensors, current sensors, and temperature sensors. The sensors transmit various signals collected during equipment operation to a server via an industrial Ethernet network. The data is then processed by a predictive maintenance system (PMS) deployed on the server, and the processing results are written into a database to form a historical data table of equipment status.

[0027] S2, construct a device recognition module, using head-mounted AR glasses (such as Epson Moverio BT-45CS) as the front-end acquisition and display terminal, equipped with an autofocus camera; the AR glasses push real-time video streams to a remote server via the RTMP protocol, with push parameters set as follows: resolution 640×480, frame rate 30fps, H.264 encoding, bitrate 1200kbps; the server deploys a pre-trained YOLOv11 object detection model, with a model input size of 640×640 and outputs the device category and bounding box pixel coordinates; after receiving the video frame, the server calls the YOLOv11 model for forward inference, returning the device category with the highest confidence and its bounding box center point coordinates (x1, y1).

[0028] S3, Data Query and Fusion: Based on the device category obtained in S2, the server queries the database for the latest status record corresponding to that device, including alarm level, maintenance recommendations, etc. The server serializes the query results into JSON format, with fields including: device name, alarm level, RMS, peak value, maintenance recommendations, sampling time, etc. Then, the server uses WebSocket full-duplex communication to exchange the JSON data with the bounding box coordinates. It will also be pushed to the AR glasses client.

[0029] S4. Construct a data visualization module. After receiving JSON data, the AR glasses client parses out the device status information and bounding box coordinates. The client calls the built-in 2D homogeneous coordinate transformation matrix H to map the image coordinates (x1, y1) to the AR display coordinates (x2, y2). The client draws a virtual rectangle centered at (x2, y2) on the transparent display screen of the AR glasses and overlays the device name, alarm level, key indicators, and maintenance suggestions on top of it.

[0030] The S5 completes a full intelligent maintenance interaction process. First, the user wears AR glasses and enters the inspection area, pointing their gaze at the target device. Then, the system automatically triggers the status acquisition, device identification, data query, and AR visualization processes in sequence, without requiring any manual operation from the user. Next, the user directly sees the overlaid device status information and maintenance suggestions in their field of vision and performs corresponding maintenance actions accordingly. Finally, when the user moves their gaze away, the virtual information automatically disappears, and the system waits for the next device to enter the field of vision to complete the next round of inspection.

[0031] See attached document Figure 3 and attached Figure 4 A real-time AR tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation is proposed. Specifically, the two-dimensional coordinate transformation matrix calibration method establishes the mapping relationship between the YOLOv11 detection frame coordinates and the AR glasses display coordinates through the following steps.

[0032] Step 1: Prepare a calibration plate with a rectangular pattern measuring 60×60mm. Fix the calibration plate to ensure no relative displacement during calibration. Figure 4 As shown; Step 2: Put the AR glasses on the operator's head, adjust the headband tightness, and align the center of the display screen with the horizontal plane of the eyes; start the client program on the glasses, enter the "coordinate calibration" mode, and a default 400×400 pixel blue preset frame will appear in the center; Step 3: The operator moves their head so that the preset box and the black rectangle on the calibration board are visually completely aligned (the left and right eyes judge separately and the average position is taken), and records the pixel coordinates of the rectangular pattern on the calibration board detected by the YOLOv11 model (the P1 and P2 coordinates displayed on the AR glasses are the image pixel coordinates (x1, y1) of the rectangular box returned by YOLOv11 after performing object detection).

[0033] Step 4: Calculate the homogeneous transformation matrix. By comparing the coordinates (x1, y1) of the rectangular pattern detected by YOLOv11 with the coordinates (x2, y2) of the preset rectangle on the Moverio display, establish the mapping relationship between the two sets of points. Let the transformation matrix be: in, , They are respectively shaft and Scaling factor for the direction of the axis. , These represent the translation distances along the x and y axes, respectively. Substituting the image pixel coordinates output by the YOLOv11 model into the transformation matrix, we convert them into pixel coordinates for display on the AR glasses: See attached document Figure 5 To verify the proposed real-time AR tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation, a prototype system was deployed in a mechanical equipment condition monitoring laboratory. Comparative inspection tests were conducted on five types of experimental equipment, with 51 participants. The results are as follows: Efficiency: Compared with the traditional manual inspection method of collecting and analyzing data using vibration meters, this system reduces the time for a single equipment inspection and diagnosis from an average of 186.4 seconds to about 4.5 seconds, improving efficiency by more than 40 times.

[0034] Accuracy: The system's data-driven diagnostic results are highly consistent with expert judgments, reducing subjective human error.

[0035] Usability: The survey showed that the vast majority of users found the system easy to use and user-friendly. AR visualization effectively reduced cognitive load and increased confidence and security in maintenance work.

[0036] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A real-time AR tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation, characterized in that, Includes the following steps: S1: Status monitoring, which collects operating data of mechanical equipment through industrial sensors. The industrial sensors are deployed in key parts of the mechanical equipment such as gearboxes, bearing housings, and reducers. The collected operating data mainly includes acceleration and current data. The operating data is automatically processed to evaluate the health status of the equipment, and the processed equipment status indicators are stored in the database. S2: Device identification, used to collect image data of mechanical equipment through the camera of AR wearable device, and transmit the image data to the server; S3: The image data is processed by the YOLOv11 target detection model deployed on the server to achieve accurate detection of the target device and output the device pixel coordinates; S4: Coordinate Transformation, used to establish a transformation matrix between the device image and the AR wearable device display unit through homogeneous coordinate transformation method, converting the device pixel coordinates output by the YOLOv11 model into the pixel coordinates of the AR device display unit, so as to realize real-time tracking and visualization of the device position; S5: Data visualization is used to call device status indicators from the database through the server, associate the device status indicators with the device identification results, and project the device status information and maintenance guidance to the user's field of vision through the display unit of the AR wearable device to realize the real-time presentation of device status information.

2. The AR real-time tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation according to claim 1, characterized in that, The status monitoring module is integrated with the predictive maintenance system (PMS) and transmits the collected operating data to the server via industrial Ethernet. The server analyzes the equipment operating status using preset fault diagnosis algorithms and anomaly detection algorithms. The equipment operating status includes health status, early warning status, and alarm status.

3. The AR real-time tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation according to claim 1, characterized in that, The device identification module transmits image data to the server via the Real-time Message Transfer Protocol (RTMP). The parameters of the image data are configured as follows: resolution 640×480 pixels, frame rate 30fps, H.264 encoding format, and bit rate 1200kbps. The server deploys a YOLOv11 target detection model trained on an industrial equipment dataset to process the image data in real time. It achieves accurate detection of target devices through multi-scale feature fusion and spatial attention mechanism, and outputs the device category, confidence score, and device pixel coordinates.

4. The AR real-time tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation according to claim 1, characterized in that, The coordinate transformation matrix is ​​established in the following manner, specifically including the following steps: Step 1: Design a calibration board with a rectangular pattern, detect the rectangular pattern of the calibration board using the trained YOLOv11 model, and record its pixel coordinates in the image; Step 2: Preset a rectangular frame on the display unit of the AR wearable device, manually adjust the preset rectangular frame to align with the rectangular pattern on the calibration board, and record the pixel coordinates of the preset rectangular frame; Step 3: Based on the pixel coordinates of the rectangular pattern on the calibration board and the pixel coordinates of the preset rectangular frame, calculate the coordinate transformation matrix through translation and scaling transformations. The coordinate transformation matrix The expression is: YOLOv11 output pixel P img : P img =[xcyc1] H The corresponding AR display coordinates are: P ar =HP img in, , They are respectively shaft and Scaling factor for the direction of the axis. , They are respectively shaft and The translation distance along the axis is used to substitute the device pixel coordinates output by the YOLOv11 model into the transformation matrix, which is then converted into the pixel coordinates of the AR device display unit, thereby realizing real-time tracking and visualization of the device position.

5. The AR real-time tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation according to claim 1, characterized in that, The server accesses device status indicators from the database via the Simple Object Access Protocol (SOAP); and serializes the device status indicators and device location data via the WebSocket protocol before transmitting them to the AR wearable device, ensuring the real-time performance and stability of data transmission.

6. The AR real-time tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation according to claim 1, characterized in that, The data visualization module presents equipment status information including equipment name, number, data acquisition time, alarm level, maintenance instructions, sampling frequency, and vibration signal indicators; the vibration signal time-domain indicators include root mean square value, kurtosis factor, skewness factor, peak value, waveform factor, impulse factor, average amplitude, and root mean square amplitude.

7. The AR real-time tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation according to claim 1, characterized in that, The vibration signal time-domain index used in the data visualization module includes the root mean square (RMS) value, which is expressed as follows: Peak value: Kurtosis: Skewness: Crest Factor: Where x i Let N be the i-th vibration acceleration sampling point, and N be the number of sampling points. X is the mean.

8. The AR real-time tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation according to claim 1, characterized in that, The state monitoring module further performs a Fast Fourier Transform (FFT) on the vibration signal: Where N is the number of sampling points, and the peak frequency f is extracted from the spectrum. peak As an indicator for equipment health diagnosis.

9. The AR real-time tracking and maintenance method for mechanical equipment based on YOLOv11 and coordinate transformation according to claim 1, characterized in that, The latency D of the AR real-time tracking system includes the image acquisition latency D. cap Network transmission delay D net and model inference delay D infer The formula for its calculation is: And it meets the real-time requirement: D≤50 ms.

Images