AR glasses for interactive inspection and interactive inspection system using same

Abstract

The invention is suitable for the technical field of AR glasses, and provides AR glasses for interactive inspection and an interactive inspection system using the AR glasses, the AR glasses for interactive inspection comprise a helmet, the helmet can be worn on the head of a user, and the bottom of the helmet is provided with a display module; the helmet comprises a hard shell, the hard shell is used for providing protection for the head of a user, an integrated sensor is arranged on the hard shell, a buffer layer is arranged at the bottom of the hard shell, the buffer layer is made of a flexible material, an attaching layer is arranged at the bottom of the buffer layer, and the attaching layer is attached to the head of the user. The system solves the problem that an operator cannot carry data and operation equipment at the same time during the inspection of the converter station, and improves the inspection efficiency and accuracy through the assistance of the AR glasses. By adjusting the head posture and the sight line deviation, the consistency of the display content and the sight line of the user is ensured, the operation precision problem caused by the sight line deviation is avoided, and the purposes of improving the working efficiency, enhancing the operation safety and optimizing the inspection experience are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of AR glasses, and more specifically, to an AR glasses for interactive inspection and an interactive inspection system using the same. Background Art

[0002] Converter stations are facilities used to convert between AC and DC in power systems, and are mainly used in high-voltage DC power transmission systems. Converter stations convert AC to DC, or vice versa. Such facilities are usually located on cross-regional or cross-national power transmission lines to help achieve long-distance and efficient power transmission. Converter stations play an important role in power systems, especially in long-distance, high-voltage power transmission, which can reduce energy losses and improve transmission efficiency.

[0003] At present, converter stations on the market need to conduct periodic inspections to ensure the safe operation of equipment, timely detect potential problems, and take appropriate repair and maintenance measures. However, in the actual inspection process, operators often face the dilemma of carrying inspection materials and operating equipment by hand at the same time. Therefore, some converter stations have introduced AR glasses and system assistance to improve inspection efficiency and accuracy. However, although this method solves some problems, there are still some challenges. However, although AR glasses have solved the problems of operating efficiency and information acquisition to a certain extent, there are still some challenges that need to be solved. First of all, due to the differences in each person's dominant eye and habitual eye, monocular AR glasses often have deviations when adapting to different users, and cannot provide an ideal display experience for all users. Although binocular AR glasses can provide a clearer view and reduce visual obstacles, due to the design of covering both eyes, it is easy to deprive operators of the ability to observe the surrounding environment, increasing potential safety hazards, especially when the inspection site environment is complex and crowded.

[0004] In addition, during the inspection of the converter station, operators often need to frequently adjust their positions or turn their heads to check equipment or details at different angles. However, due to changes in head posture, the display content of AR glasses can easily deviate from the user's line of sight, causing the display information to be misplaced or blurred, thus affecting the accuracy of the operation. Even with algorithms for head posture adjustment, it is still difficult to completely eliminate the display instability problem caused by position changes, affecting the operating experience during the inspection process.

[0005] Therefore, the present invention proposes an AR glasses for interactive inspection and an interactive inspection system using the same. Summary of the invention

[0006] In view of the deficiencies in the prior art, an object of the present invention is to provide an AR glasses for interactive inspection and an interactive inspection system using the same.

[0007] To achieve the above object, the present invention provides the following technical solutions: An interactive inspection system, comprising AR glasses, and further comprising: An image processing module, the image processing module being connected to the AR glasses, the image processing module being used to pre-process a device image captured by a capture device of the integrated sensor (111) and extract feature points or feature areas in the device image, analyze the shape and geometric information of the device by a feature extraction algorithm, then perform device identification, compare the extracted device features with a device template stored in a database, and determine the device identity; An optical character recognition module, which is used to recognize text information in the device image, including the number, label and QR code on the device; further verify the identity of the device by comparing the text information recognized by the optical character recognition with the data in the device database; The back-end database is used to store the template data of the equipment and the identification information of the equipment. The template data of the equipment includes the equipment image template, three-dimensional model, and feature descriptor; the identification information of the equipment includes the equipment number, model, status, and maintenance record; A user interaction module, which is used to provide user feedback during the device recognition process and supports voice input, so that the user can manually confirm device information or enter additional data; the user interaction module includes a visual feedback unit for enabling user interaction with the AR glasses; The data synchronization module is used to synchronize device recognition results, image data and device status to the cloud platform to ensure real-time updating of device management information.

[0008] By adopting the above technical solutions, combined with head posture tracking, equipment recognition and interaction technology, the work efficiency and accuracy of on-site staff can be greatly improved. The system's equipment information storage module can record the equipment number, model, location, operating status and maintenance records in detail, providing data support for real-time management of equipment. The equipment update module is responsible for receiving and processing equipment images and recognition results from AR glasses, updating equipment information in a timely manner, and ensuring the accuracy of data. The system uses head posture tracking and line of sight consistency algorithms to ensure that the display content of AR glasses is synchronized with the user's line of sight, and can always provide accurate visual feedback regardless of whether the user chooses to place AR glasses in the left or right eye position. The user interaction module includes functions such as voice input, visual feedback and status indication. Users can interact with the system through voice commands to obtain equipment recognition results, equipment status information and system operation instructions in real time, ensuring the convenience and accuracy of operation. In addition, the system can automatically generate equipment usage reports and maintenance reports to help staff efficiently manage equipment and improve the accuracy and work efficiency of inspection tasks.

[0009] The present invention is further configured as follows: the image processing module further comprises: Image preprocessing unit, used to perform operations such as denoising, enhancement, and contrast adjustment of device images to ensure that the quality of captured images meets recognition requirements; A feature extraction unit is used to extract key feature points of the device from the preprocessed device image, including geometric shape, surface texture, color distribution, etc.; An image matching unit, which is used to compare the extracted feature points with the device template in the database and determine the identity of the device using a feature matching algorithm; The positioning auxiliary unit, based on the positioning information of the glasses and combined with the device template information, improves the accuracy of device model determination, especially when similar devices exist.

[0010] The present invention is further configured as follows: the back-end database is connected to the remote server via a cloud platform, and the cloud platform includes: The equipment information storage module is used to store detailed information of the equipment in the battery swap station, including equipment number, model, location, operating status, and maintenance records; The device update module is used to receive and store device images, recognition results and device status collected by AR glasses, and update device information in real time based on device and manual feedback; The equipment management module is used for querying, retrieving and generating reports on equipment information, and supports the generation of equipment trial reports and equipment maintenance reports based on equipment identification results.

[0011] The present invention is further configured as follows: the user interaction module comprises: The voice input unit is used to receive voice commands from the user to interact with the AR glasses and execute functions such as device confirmation and operation instructions; The visual feedback unit is used to provide real-time feedback to the user through the AR glasses display, including device recognition results, device status, and system operation instructions; A status indication unit, used to generate a highlighted indication, graphic annotation or text information of the device on the display screen of the AR glasses to guide the user to operate or perform device detection; The interactive verification unit is used to verify the commands issued by the user to ensure the effectiveness and accuracy of the user interaction and avoid misoperation.

[0012] The present invention is further configured as follows: the visual feedback unit adjusts the position of the displayed content in real time by combining eye tracking, head posture monitoring and the relative position information of the AR glasses themselves with multiple input sources to ensure that the displayed information is consistent with the user's line of sight and perform visual adjustment. The specific steps are: S1, obtain eye tracking data through the built-in eye tracker of AR glasses to obtain the line of sight direction g; S2. Get the head posture, which uses the rotation matrix R head express; S3. Obtain the relative position information R of the AR glasses user ; S4, calculating the adjustment amount ΔP of the displayed content according to the above information; S5. Adjust the display content adjustment position according to the adjustment amount ΔP of the display content. The calculation formula is: P final =P screen +ΔP; Among them, P final is the final display position, P screen is the initial display position, and ΔP is the adjustment amount of the display content.

[0013] The present invention is further configured as follows: the visual feedback unit obtains the user's head posture through a sensor built into the AR glasses, and adaptively adjusts the AR glasses according to the head posture, so that the center position of the user's head always falls corresponding to the center position of the display part of the AR glasses; Among them, the built-in sensors of the AR glasses include an accelerometer and a gyroscope, and the visual feedback unit calculates the head posture according to the accelerometer, the gyroscope and the current angle feedback of the AR glasses themselves.

[0014] The present invention is further configured as follows: the head posture calculation only calculates the head rotation, uses Euler angles to express the head rotation, and uses the rotation matrix R head Describes the head orientation, where the rotation matrix R head Including the yaw rotation matrix R yaw , pitch rotation matrix R pitch , roll rotation matrix R roll ; The calculation formula of the yaw rotation matrix is: ; The calculation formula of the pitch rotation matrix is: ; The calculation formula of the roll rotation matrix is: ; The calculation formula of the rotation matrix is: R head =R yaw ·R pitch ·R roll； The rotation described by the rotation matrix is ​​the rotation of the head relative to the reference coordinate system, θ yaw is the yaw angle, θ pitch is the pitch angle, θroll is the roll angle.

[0015] The present invention also provides the following technical solution: an AR glasses for interactive inspection, comprising a helmet, the helmet can be worn on the head of a user, and a display module is arranged at the bottom of the helmet; The helmet includes a hard shell, which is used to provide protection for the user's head. An integrated sensor is arranged on the hard shell. A buffer layer is arranged at the bottom of the hard shell. The buffer layer is arranged with a flexible material and is used to alleviate impact force. A fitting layer is arranged at the bottom of the buffer layer, and the fitting layer is arranged to fit the user's head.

[0016] The present invention is further configured as follows: the bonding layer includes a center plate, the center plate is configured as a circular structure, a bonding belt is arranged around the center plate, an adjustment belt for adjustment is installed at the bottom of the bonding belt, a mounting strip is arranged on one side of the adjustment belt, and the mounting strip provides an installation environment for the display module.

[0017] The present invention is further configured as follows: the display module comprises an arc-shaped slide rail mounted on the mounting bar, a slider is slidably connected to the arc-shaped slide rail, the slider can slide relative to the arc-shaped slide rail, and a display assembly is connected to the bottom of the slider.

[0018] By adopting the above technical solution, the user's head posture is obtained in real time and adaptively adjusted through the visual feedback unit and the built-in sensors of the AR glasses (such as accelerometers and gyroscopes), ensuring that the AR glasses display content is always aligned with the user's line of sight. The head posture calculation method is mainly aimed at the rotation of the head, and Euler angles (yaw, pitch and roll angles) are used to describe the multi-axis rotation of the head. Through the yaw, pitch and roll rotation matrices, the system can accurately model the rotational movement of the head. The yaw rotation matrix calculates the rotation around the Z axis, the pitch rotation matrix calculates the rotation around the X axis, and the roll rotation matrix calculates the rotation around the Y axis. Finally, by combining these rotation matrices, the overall rotation matrix of the head is obtained to describe the rotation direction of the head. This method ensures that the display part of the AR glasses is always accurately aligned with the user's line of sight in different postures, optimizing the AR interactive experience.

[0019] In summary, the present application includes at least one of the following beneficial technical effects: 1. This application greatly improves the work efficiency and accuracy of on-site staff by combining head posture tracking, equipment recognition and interaction technology. The system's equipment information storage module can record the equipment number, model, location, operating status and maintenance records in detail, providing data support for real-time management of the equipment. The equipment update module is responsible for receiving and processing equipment images and recognition results from AR glasses, updating equipment information in a timely manner, and ensuring the accuracy of the data. The system uses head posture tracking and line of sight consistency algorithms to ensure that the display content of AR glasses is synchronized with the user's line of sight, and can always provide accurate visual feedback regardless of whether the user chooses to place the AR glasses in the left or right eye position. The user interaction module includes functions such as voice input, visual feedback and status indication. Users can interact with the system through voice commands to obtain equipment recognition results, equipment status information and system operation instructions in real time to ensure the convenience and accuracy of operation. In addition, the system can automatically generate equipment usage reports and maintenance reports to help staff efficiently manage equipment and improve the accuracy and work efficiency of inspection tasks.

[0020] 2. This application uses the visual feedback unit and the built-in sensors of the AR glasses (such as accelerometers and gyroscopes) to obtain the user's head posture in real time and make adaptive adjustments to ensure that the AR glasses display content is always aligned with the user's line of sight. The head posture calculation method is mainly aimed at the rotation of the head, and Euler angles (yaw, pitch and roll angles) are used to describe the multi-axis rotation of the head. Through the yaw, pitch and roll rotation matrices, the system can accurately model the rotational movement of the head. The yaw rotation matrix calculates the rotation around the Z axis, the pitch rotation matrix calculates the rotation around the X axis, and the roll rotation matrix calculates the rotation around the Y axis. Finally, by combining these rotation matrices, the overall rotation matrix of the head is obtained to describe the rotation direction of the head. This method ensures that the display part of the AR glasses is always accurately aligned with the user's line of sight in different postures, optimizing the AR interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the structure of AR glasses for interactive inspection in the present invention.

[0022] Figure 2 for Figure 1 Schematic diagram of the explosion.

[0023] Figure 3 for Figure 2 Schematic diagram of the explosion.

[0024] Description of reference numerals: 1. helmet; 11. hard shell; 111. integrated sensor; 12. buffer layer; 13. fitting layer; 131. center plate; 132. fitting belt; 133. adjustment belt; 134. mounting strip; 2. Display module; 21. Arc slide rail; 22. Slider; 23. Display assembly. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] It should be noted that, unless otherwise specified, all technical and scientific terms used in this application have the same meanings as commonly understood by ordinary technicians in the technical field to which this application belongs.

[0027] See also Figure 1-3 , the present invention provides the following technical solutions: For example, see Figure 1-3 , an AR glasses for interactive inspection, comprising a helmet 1, the helmet 1 can be worn on the head of a user, and a display module is arranged at the bottom of the helmet 1; See also Figure 1 The helmet 1 includes a hard shell 11, which is used to provide protection for the user's head. An integrated sensor 111 is arranged on the hard shell 11, and a buffer layer 12 is arranged at the bottom of the hard shell 11. The buffer layer 12 is adapted to the shape of the hard shell 11, and the buffer layer 12 is set to a flexible material. The buffer layer 12 is used to alleviate the impact force, providing a double guarantee for the user's safety, ensuring that after the hard shell 11 is damaged, the buffer layer 12 can prevent the broken hard shell 11 from hurting the user, and a fitting layer 13 is arranged at the bottom of the buffer layer 12, and the fitting layer 13 is set to fit the user's head.

[0028] It should be noted that the integrated sensor 111 is arranged in the middle of the hard shell 11 to ensure that it can maintain balance in the overall structure of the device and obtain the best perception effect. The integrated sensor 111 is a highly integrated module, which contains a camera component, an accelerometer, and a gyroscope for obtaining external environmental information; the accelerometer can sense the linear motion of the head, and the changes in the acceleration direction and amplitude are clear at a glance, while the gyroscope captures the rotational motion of the head by sensing the angular velocity; through the cooperation of the two, the position and posture of the head can be tracked in real time.

[0029] See also Figure 2 The fitting layer 13 includes a central plate 131, which is arranged in a circular structure. A fitting belt 132 is arranged around the central plate 131. An adjustment belt 133 for adjustment is installed at the bottom of the fitting belt 132. The adjustment belt 133 can be adjusted according to the user's head circumference to ensure the user's wearing comfort and prevent it from falling off. A mounting strip 134 is arranged on one side of the adjustment belt 133. The mounting strip 134 is arranged in a symmetrical position of the adjustment belt. The mounting strip 134 provides an installation environment for the display module.

[0030] See also Figure 3 The display module includes an arc-shaped slide rail 21 installed on the mounting bar 134, and a slider 22 is slidably connected to the arc-shaped slide rail 21. The slider 22 can slide relative to the arc-shaped slide rail 21. A display component 23 is connected to the bottom of the slider 22. The sliding of the slider 22 can drive the display component 23 to slide, thereby adjusting the display position.

[0031] AR glasses for interactive inspection need to be used in conjunction with the interactive inspection system, which includes: The image processing module is connected to the AR glasses. The image processing module is used to pre-process the captured device image and extract feature points or feature areas in the device image, analyze the shape and geometric information of the device through a feature extraction algorithm, and then perform device identification, compare the extracted device features with the device template stored in the database, and determine the device identity.

[0032] Optical character recognition module, the optical character recognition module is used to recognize text information in the device image, including the number, label and QR code on the device; by comparing the text information of optical character recognition with the data in the device database, the identity of the device is further verified.

[0033] The back-end database is used to store the template data of the equipment and the identification information of the equipment. The template data of the equipment includes the equipment image template, three-dimensional model, and feature descriptor; the identification information of the equipment includes the equipment number, model, status, and maintenance record.

[0034] The user interaction module is used to provide user feedback during the device recognition process and supports voice input, so that users can manually confirm device information or enter additional data; the user interaction module includes a visual feedback unit for realizing user interaction with AR glasses.

[0035] The data synchronization module is used to synchronize device recognition results, image data and device status to the cloud platform to ensure real-time updating of device management information.

[0036] The image processing module includes: Image preprocessing unit, used to perform operations such as denoising, enhancement, and contrast adjustment of device images to ensure that the quality of captured images meets recognition requirements; A feature extraction unit is used to extract key feature points of the device from the preprocessed device image, including geometric shape, surface texture, color distribution, etc.; An image matching unit, which is used to compare the extracted feature points with the device template in the database and determine the identity of the device using a feature matching algorithm; The positioning auxiliary unit, based on the positioning information of the glasses and combined with the device template information, improves the accuracy of device model determination, especially when similar devices exist.

[0037] The backend database is connected to the remote server through the cloud platform, which includes: The equipment information storage module is used to store detailed information about the equipment in the battery swap station, including equipment number, model, location, operating status, and maintenance records. Each device has a unique number in the system to identify the device. The equipment number will be associated with the equipment model, location, operating status, and other information to ensure that relevant equipment information can be accurately located and obtained during inquiries. The equipment model includes the equipment's technical specifications, manufacturer, and production date. This information will be helpful for subsequent equipment maintenance and equipment life cycle management.

[0038] The operating status of the equipment is the current working condition of the equipment, including whether it is operating normally, fault information, downtime, etc. This information is the basis for equipment monitoring and maintenance, and helps to determine whether the equipment needs to be repaired or replaced. Equipment maintenance records store the historical maintenance records of the equipment, including detailed information on repairs, maintenance, upgrades, and other operations. Maintenance records are essential for equipment managers to assess equipment health, develop maintenance plans, and determine equipment life.

[0039] The equipment update module is used to receive and store equipment images, recognition results and equipment status collected by AR glasses, and to update equipment information in real time based on equipment and manual feedback; this module uses AR glasses to capture images of equipment in the battery swap station in real time, and uses advanced image recognition technology to identify the equipment, including key information such as the appearance, number, model, etc. of the equipment, to ensure that the equipment information is always consistent with the actual equipment. When the equipment status changes or a failure occurs, the system will automatically update the equipment status information. In addition, the equipment update module also supports manual feedback. When the system recognition results are inaccurate or the equipment status is abnormal, the user can correct the equipment information through manual input. In this way, the system ensures the real-time update of the equipment information database, allowing the system to respond to the operation of the equipment at any time and adjust the equipment management strategy when necessary.

[0040] The equipment management module is used for querying, retrieving and generating reports on equipment information, and supports the generation of equipment trial reports and equipment maintenance reports based on equipment identification results.

[0041] The user interaction module is an important bridge between the system and the user, supporting users to interact with the system in various ways to improve user experience and operational efficiency. The user interaction module includes: The voice input unit is used to receive voice commands from users to interact with AR glasses and execute functions such as device confirmation and operation instructions. It eliminates the need for manual operation and is suitable for scenarios where users have difficulty operating with their hands.

[0042] The visual feedback unit is used to provide real-time feedback to the user through the AR glasses display, including device recognition results, device status, and system operation instructions; A status indication unit, used to generate a highlighted indication, graphic annotation or text information of the device on the display screen of the AR glasses to guide the user to operate or perform device detection; It should be noted that part of the highlighted indication, graphic annotation or text information generated by the device on the display screen of the AR glasses is the information for feedback on the application submitted by the user himself. At the same time, the present invention is also equipped with a remote contact unit, through which the instructor far away at the back end can give guidance to the user using the AR glasses, including prompts and annotations, and the annotations can be made in the form of graphics or text. Among them, the prompts and annotations are set to a font with a transparency of 50% to avoid blocking the current information. It is further explained that the remote contact unit can directly initiate a contact application to the AR glasses, and at the same time, the user can also send a contact application to the remote contact unit through the AR glasses.

[0043] The interactive verification unit is used to verify the commands issued by the user, ensure the effectiveness and accuracy of the user interaction, and avoid misoperation. In this application, when receiving an unusual command from the user, the AR glasses will ask the user whether to execute the command, and will execute it after the user nods three times.

[0044] The visual feedback unit uses eye tracking, head posture monitoring, and the relative position information of the AR glasses combined with multiple input sources to adjust the position of the displayed content in real time to ensure that the displayed information is consistent with the user's line of sight, avoid information transmission errors caused by the offset between the displayed information and the user's line of sight, and perform visual adjustments. The specific steps are as follows: S1, obtain eye tracking data through the built-in eye tracker of AR glasses to obtain the line of sight direction g; S2. Get the head posture, which uses the rotation matrix R head express; S3. Obtain the relative position information R of the AR glasses user ; S4, calculating the adjustment amount ΔP of the displayed content according to the above information; S5. Adjust the display content adjustment position according to the adjustment amount ΔP of the display content. The calculation formula is: P final =P screen +ΔP; Among them, P finalis the final display position, P screen is the initial display position, and ΔP is the adjustment amount of the display content.

[0045] It should be noted that the ΔP calculation process is as follows: A1. Calculate the adjustment amount ΔP based on the line of sight direction gaze First, P screen is the initial display position (P screen The default position is set for the first time wear, and the current position is set for non-first wear). The sight direction g defines the position where the display content should be displayed, and the adjustment amount based on the sight direction is recorded as ΔP gaze , then the adjustment amount ΔP based on the line of sight direction gaze The calculation formula is: ΔP gaze =g·α gaze Among them, α gaze is a coefficient that indicates the influence of sight. This coefficient can be adjusted according to the needs of actual applications to control the relative position of the display content and sight.

[0046] A2. Calculate the adjustment amount ΔP based on the head posture rotation ; Head rotation will cause rotation adjustment of the display content position. We use the display content position P screen Apply the rotation matrix R head To calculate the new display content position: ΔP rotation =R head ·(P screen -P user ), P user is the reference position of the user's eyes.

[0047] A3. Calculate the comprehensive adjustment amount ΔP. The comprehensive adjustment amount ΔP will comprehensively consider the line of sight direction, head posture and the relative position information of the AR glasses themselves, and perform weighted combination of various adjustment amounts to obtain the final display content adjustment amount: ΔP=ΔP gaze +ΔP rotation It should be noted that the relative position information of the AR glasses is the relative position of the display module relative to the helmet 1. The relative position can be used to analyze whether the user places the display module on the left eye or the right eye, as well as the specific position of the display module.

[0048] The visual feedback unit obtains the user's head posture through the built-in sensors of the AR glasses, and adaptively adjusts the AR glasses according to the head posture, so that the center position of the user's head always corresponds to the center position of the display part of the AR glasses.

[0049] Among them, the built-in sensors of AR glasses include accelerometers and gyroscopes. The visual feedback unit calculates the head posture based on the accelerometer, gyroscope and the current angle feedback of the AR glasses themselves.

[0050] Since the rotation of the head will cause the line of sight and the position of the displayed content to change, in the calculation of the head posture, it is necessary not only to describe the rotation in a single direction, but also to model the multi-axis rotation. The user may rotate on three axes: pitch, yaw, and roll. The rotation matrix allows us to describe complex multi-axis rotations by combining multiple rotation matrices. Therefore, the present invention uses the rotation matrix to process the head posture change.

[0051] The head posture calculation method only calculates the head rotation and uses Euler angles to express the head rotation. It should be noted that Euler angles are a common method for describing rigid body rotation, which consists of three angles: Yaw Angle: The angle of rotation about the z-axis. The yaw angle describes the rotation of an object about its vertical axis (usually the axis on the ground).

[0052] Pitch Angle: The angle of rotation around the Y axis. The pitch angle describes the rotation of an object around its horizontal axis (the axis perpendicular to the direction of travel).

[0053] Roll Angle: The angle of rotation around the X axis. The roll angle describes the rotation of an object around its forward-direction axis.

[0054] And use the rotation matrix R head Describes the head orientation, the rotation matrix R head Is a 3*3 matrix, where the rotation matrix R head Including the yaw rotation matrix R yaw (Calculate the yaw angle), pitch rotation matrix R pitch (Calculate pitch angle), roll rotation matrix R roll (Calculate the roll angle); The calculation formula of the yaw rotation matrix is: ; In yaw rotation, the Z axis is the axis of rotation, so it primarily affects the directions of the x and y axes.

[0055] The calculation formula of the pitch rotation matrix is: ; Pitch rotation causes the head to rotate around the horizontal axis (X axis), and is often used to describe the action of the user nodding his head up and down. In this application, it is applied to the interactive verification unit.

[0056] The calculation formula of the roll rotation matrix is: ; Roll rotation describes the rotation of the head around the front-to-back axis (Y axis) and is often used to describe the action of the user's head tilting left or right.

[0057] The calculation formula of the rotation matrix is: R head =R yaw ·R pitch ·R roll ; The rotation described by the rotation matrix is ​​the rotation of the head relative to the reference coordinate system, θ yaw is the yaw angle, the angle of rotation around the vertical axis (Z axis) of the object, θ pitch is the pitch angle, the angle of rotation around the object's lateral axis (Y axis), θ roll The roll angle is the angle of rotation around the object's forward direction axis (X axis).

[0058] This application greatly improves the work efficiency and accuracy of on-site staff by combining head posture tracking, equipment recognition and interaction technology. The system's equipment information storage module can record the equipment number, model, location, operating status and maintenance records in detail, providing data support for real-time management of the equipment. The equipment update module is responsible for receiving and processing equipment images and recognition results from AR glasses, updating equipment information in a timely manner, and ensuring the accuracy of the data. The system uses head posture tracking and line of sight consistency algorithms to ensure that the display content of AR glasses is synchronized with the user's line of sight, and can always provide accurate visual feedback regardless of whether the user chooses to place the AR glasses in the left or right eye position. The user interaction module includes functions such as voice input, visual feedback and status indication. Users can interact with the system through voice commands to obtain equipment recognition results, equipment status information and system operation instructions in real time to ensure the convenience and accuracy of operation. In addition, the system can automatically generate equipment usage reports and maintenance reports to help staff efficiently manage equipment and improve the accuracy and work efficiency of inspection tasks.

[0059] The device embodiments described above are merely illustrative and are not all embodiments. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms. All other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

Claims

1. An interactive inspection system, comprising AR glasses for interactive inspection, characterized in that: Also includes: An image processing module, the image processing module being connected to the AR glasses, the image processing module being used to pre-process a device image captured by a capture device of the integrated sensor (111) and extract feature points or feature areas in the device image, analyze the shape and geometric information of the device by a feature extraction algorithm, then perform device identification, compare the extracted device features with a device template stored in a database, and determine the device identity; An optical character recognition module, which is used to recognize text information in the device image, including the number, label and QR code on the device; further verify the identity of the device by comparing the text information recognized by the optical character recognition with the data in the device database; The back-end database is used to store the template data of the equipment and the identification information of the equipment. The template data of the equipment includes the equipment image template, three-dimensional model, and feature descriptor; the identification information of the equipment includes the equipment number, model, status, and maintenance record; A user interaction module, which is used to provide user feedback during the device recognition process and supports voice input, so that the user can manually confirm device information or enter additional data; the user interaction module includes a visual feedback unit for enabling user interaction with the AR glasses; The data synchronization module is used to synchronize device recognition results, image data and device status to the cloud platform to ensure real-time updating of device management information.

2. An interactive inspection system according to claim 1, characterized in that: The image processing module further comprises: Image preprocessing unit, used to perform operations such as denoising, enhancement, and contrast adjustment of device images to ensure that the quality of captured images meets recognition requirements; A feature extraction unit is used to extract key feature points of the device from the preprocessed device image, including geometric shape, surface texture, color distribution, etc.; An image matching unit, which is used to compare the extracted feature points with the device template in the database and determine the identity of the device using a feature matching algorithm; The positioning auxiliary unit, based on the positioning information of the glasses and combined with the device template information, improves the accuracy of device model determination, especially when similar devices exist.

3. The interactive inspection system according to claim 1, characterized in that: The backend database is connected to the remote server via a cloud platform, and the cloud platform includes: The equipment information storage module is used to store detailed information of the equipment in the battery swap station, including equipment number, model, location, operating status, and maintenance records; The device update module is used to receive and store device images, recognition results and device status collected by AR glasses, and update device information in real time based on device and manual feedback; The equipment management module is used for querying, retrieving and generating reports on equipment information, and supports the generation of equipment trial reports and equipment maintenance reports based on equipment identification results.

4. The interactive inspection system according to claim 1, characterized in that: The user interaction module comprises: The voice input unit is used to receive voice commands from the user to interact with the AR glasses and execute functions such as device confirmation and operation instructions; The visual feedback unit is used to provide real-time feedback to the user through the AR glasses display, including device recognition results, device status, and system operation instructions; A status indication unit, used to generate a highlighted indication, graphic annotation or text information of the device on the display screen of the AR glasses to guide the user to operate or perform device detection; The interactive verification unit is used to verify the commands issued by the user to ensure the effectiveness and accuracy of the user interaction and avoid misoperation.

5. The interactive inspection system according to claim 4, characterized in that: The visual feedback unit adjusts the position of the displayed content in real time by combining multiple input sources through eye tracking, head posture monitoring and the relative position information of the AR glasses themselves, ensuring that the displayed information is consistent with the user's line of sight and performing visual adjustments. The specific steps are as follows: S1, obtain eye tracking data through the built-in eye tracker of AR glasses to obtain the line of sight direction g; S2. Get the head posture, which uses the rotation matrix R head express; S3. Obtain the relative position information R of the AR glasses user ; S4, calculating the adjustment amount ΔP of the displayed content according to the above information; S5. Adjust the display content adjustment position according to the adjustment amount ΔP of the display content. The calculation formula is: P final =P screen +ΔP; Among them, P final is the final display position, P screen is the initial display position, and ΔP is the adjustment amount of the display content.

6. The interactive inspection system according to claim 1, characterized in that: The visual feedback unit obtains the user's head posture through the built-in sensor of the AR glasses, and adaptively adjusts the AR glasses according to the head posture, so that the center position of the user's head always falls corresponding to the center position of the display part of the AR glasses; Among them, the built-in sensors of the AR glasses include an accelerometer and a gyroscope, and the visual feedback unit calculates the head posture according to the accelerometer, the gyroscope and the current angle feedback of the AR glasses themselves.

7. An interactive inspection system according to claim 6, characterized in that: The head posture calculation only calculates the head rotation, using Euler angles to express the head rotation, using the rotation matrix R head Describes the head orientation, where the rotation matrix R head Including the yaw rotation matrix R yaw , pitch rotation matrix R pitch , roll rotation matrix R roll ; The calculation formula of the yaw rotation matrix is: ； The calculation formula of the pitch rotation matrix is: ； The calculation formula of the roll rotation matrix is: ； The calculation formula of the rotation matrix is: R head =R yaw ·R pitch ·R roll ; The rotation described by the rotation matrix is ​​the rotation of the head relative to the reference coordinate system, θ yaw is the yaw angle, θ pitch is the pitch angle, θ roll is the roll angle.

8. An AR glasses for interactive inspection, applied to the interactive inspection system according to any one of claims 1 to 7, characterized in that: The helmet (1) comprises a helmet (1), which can be worn on the head of a user, and a display module (2) is arranged at the bottom of the helmet (1); The helmet (1) comprises a hard shell (11), the hard shell (11) being used to provide protection for a user's head, an integrated sensor (111) being arranged on the hard shell (11), a buffer layer (12) being arranged at the bottom of the hard shell (11), the buffer layer (12) being arranged to be made of a flexible material, the buffer layer (12) being used to mitigate impact force, and a fitting layer (13) being arranged at the bottom of the buffer layer (12), the fitting layer (13) being arranged to fit the user's head.

9. The AR glasses for interactive inspection according to claim 8, characterized in that: The laminating layer (13) comprises a central plate (131), the central plate (131) being arranged in a circular structure, a laminating belt (132) being arranged around the central plate (131), an adjusting belt (133) for adjustment being installed at the bottom of the laminating belt (132), a mounting strip (134) being arranged on one side of the adjusting belt (133), the mounting strip (134) providing a mounting environment for the display module (2).

10. The AR glasses for interactive inspection according to claim 9, characterized in that: The display module (2) comprises an arc-shaped slide rail (21) mounted on a mounting bar (134), a slider (22) being slidably connected to the arc-shaped slide rail (21), the slider (22) being capable of sliding relative to the arc-shaped slide rail (21), and a display assembly (23) being connected to the bottom of the slider (22).

Images