A multi-functional detection, recognition and interaction system based on AR glasses and its usage method

Abstract

The present invention relates to a multi-functional detection, recognition and interaction system based on an AR glasses, which includes a headband wearing member and a device host. The back plate of the device host is connected to the front part of the headband wearing member. A dual-diffraction optical machine assembly and a gesture recognition assembly are provided at the bottom of the device host. The dual-diffraction optical machine assembly includes an image output optical machine and a waveguide diffraction lens. The waveguide diffraction lens is used to image the image source into the left and right eye visual fields; the gesture recognition assembly is used to capture and recognize the gestures of the wearer; a camera and a TOF sensor are provided in the middle of the front side of the device host. By using the image output optical machine and the waveguide diffraction lens, the present invention can provide clearer images. The waveguide diffraction lens enables a thinner and lighter design and helps to naturally superimpose virtual information onto the user's field of view without causing obvious visual distortion; the built-in gesture recognition assembly enables the user to send instructions through simple gestures; the equipped camera and TOF sensor are used in combination to perform functions such as precise spatial positioning and object distance measurement.

Description

Technical Field

[0001] The present invention relates to the technical field of augmented reality. Specifically, it relates to a multifunctional detection, recognition and interaction system based on AR glasses and its usage method. Background Art

[0002] In an industrial environment, augmented reality (AR) glasses, as a tool capable of providing real-time information and guidance, have received extensive attention. Some industrial working environments require wearing safety helmets, such as power inspections in power plants, fire rescue at fire scenes, and surveying operations at construction sites; some industrial working environments do not require wearing safety helmets, such as electronic maintenance in electronic assembly workshops and home appliance repairs.

[0003] Existing AR glasses products usually adopt a relatively traditional glasses structure design. The electronic components for displaying virtual images, such as displays, optical components, processors, and other electronic modules, are mainly concentrated in the part between the two frames, and the problem of weight distribution is not well solved, resulting in fatigue or discomfort after long-term wearing, especially with greater pressure on the nose bridge and ears. Moreover, the AR glasses with a conventional glasses structure have less configurable space, resulting in only basic display functions and lacking advanced interaction means such as gesture recognition and voice control, which limits their application potential in complex scenarios such as industry.

[0004] The above problems are worthy of solution. Summary of the Invention

[0005] In order to overcome the deficiencies of the existing technology, the present invention provides a multifunctional detection, recognition and interaction system based on AR glasses and its usage method.

[0006] The technical solution of the present invention is as follows:

[0007] The present invention provides a multifunctional detection, recognition and interaction system based on AR glasses, including a headband wearing member and a device host. The back plate of the device host is connected to the front lower edge of the headband wearing member. A dual-diffraction optical machine component and a gesture recognition component are provided at the bottom of the device host. The dual-diffraction optical machine component includes an image output optical machine and a waveguide diffraction lens. The image output optical machine is used to output an image source to the waveguide diffraction lens, and the waveguide diffraction lens is used to image the image source into the left and right eye visual field areas of the wearer; the gesture recognition component is used to capture and recognize the gestures of the wearer; a camera and a TOF sensor are provided in the middle of the front side of the device host. The camera is used to capture the real-scene image of the surrounding environment, and the TOF sensor is used to measure the distance between the object and the TOF sensor in real time;

[0008] The gesture recognition component captures and recognizes the gestures of the wearer, including:

[0009] S1. Read in the pre-trained palm detection model and hand keypoint detection model;

[0010] S2. The gesture recognition lens captures the hand image;

[0011] S3. Use the palm detection model to locate the palm area in the hand image;

[0012] S4. Use the hand keypoint detection model to obtain the keypoint positions of the hand;

[0013] S5. Utilize the position information of the keypoints and calculate the finger angles through an algorithm;

[0014] S501. Calculate the first vector from the wrist to the finger root;

[0015] S502. Calculate the second vector from the finger root to the fingertip;

[0016] S503. Calculate the angle between the first vector and the second vector;

[0017] S6. Compare the calculated finger angles with the preset threshold to determine the gesture type.

[0018] In the present invention according to the above solution, infrared LED fill lights are provided on both sides of the camera.

[0019] In the present invention according to the above solution, the working mainboard of the device host integrates a processing unit and a storage unit. The processing unit is used to analyze the environment according to the real scene and spatial positioning data obtained by the camera and / or sensor of the AR glasses, and automatically calculate the corresponding positions for placing virtual content. It is also used to retrieve the display content from the storage unit and display it at the corresponding positions.

[0020] In the present invention according to the above solution, a thermal imaging lens assembly is provided on the side of the device host. The thermal imaging lens assembly is provided with a manual focus ring, and the focal length of the thermal imaging lens is adjusted by rotating the manual focus ring.

[0021] In the present invention according to the above solution, the working mainboard of the device host integrates a processing unit and a storage unit. The processing unit is used to analyze the environment according to the real scene and spatial positioning data obtained by the camera and / or sensor of the AR glasses, and automatically calculate the corresponding positions for placing virtual content. It is also used to retrieve the display content from the storage unit and display it at the corresponding positions.

[0022] In the present invention according to the above solution, the working mainboard of the device host integrates a positioning module and an inertial measurement module. The positioning is used to obtain the current position of the user, and the inertial measurement module is used to detect the motion state of the user.

[0023] The present invention according to the above solution further includes a control host, which is electrically connected to the device host through a data cable to transmit data or send control instructions to the device host.

[0024] In the present invention according to the above solution, the control host is provided with a pickup and stereo headphones, and the main control board of the control host integrates a processor, a cellular network module, an image codec module, and an audio codec module;

[0025] The control host receives audio data streams and video data streams through 4G / 5G networks;

[0026] The audio codec module decodes the received audio data stream, converts it into a playable audio signal, and the stereo headphones play the audio;

[0027] The image codec module decodes the received video data stream, converts it into a displayable image frame, and the double-diffraction optical machine component plays the image frame.

[0028] In the present invention according to the above solution, the gesture recognition component is arranged on the right side of the bottom of the device host, and the shooting direction of its gesture recognition lens faces downward to closely capture the right hand gesture of the wearer.

[0029] The present invention also provides a usage method of a multifunctional detection and recognition interaction system based on AR glasses according to the above solution:

[0030] Fix the headband wearing part on the head, ensure that the device host is in front of the forehead and the double-diffraction optical machine component is in front of the bridge of the nose, and the waveguide diffraction lens faces the eyes directly;

[0031] Start the control host and the device host, and the system performs self-check and initialization, including the camera, infrared LED fill light, TOF sensor, gesture recognition component, and thermal imaging lens;

[0032] The camera captures the real scene image of the surrounding environment, the TOF sensor measures the distance between the object and the sensor in real time, and the system analyzes the user's environment;

[0033] Automatically calculate the position of the virtual content according to the data, retrieve the corresponding content, and superimpose the virtual image onto the user's real vision through the waveguide diffraction lens;

[0034] Adjust the focal length of the thermal imaging lens through the manual focus ring;

[0035] The thermal imaging lens component captures the thermal radiation of the object and converts it into a visual thermal image;

[0036] The gesture recognition component captures and recognizes the user's gestures, and transmits the recognized gesture signals to the processing unit to output corresponding instructions or corresponding information;

[0037] The control host communicates remotely with other control hosts or the backend platform through 4G / 5G network, including sending text messages and real-time audio and video broadcasts.

[0038] According to the present invention of the above solution, the gesture recognition component captures and recognizes the user's gestures, including:

[0039] S1. Read in the pre-trained palm detection model and hand key point detection model;

[0040] S2. The gesture recognition lens captures the hand image;

[0041] S3. Use the palm detection model to locate the palm area in the hand image;

[0042] S4. Use the hand key point detection model to obtain the key point positions of the hand;

[0043] S5. Utilize the position information of the key points to calculate the finger angles through an algorithm;

[0044] S6. Compare the calculated finger angles with the preset thresholds to determine the gesture type;

[0045] Further, the calculation of the finger angles in step 5 includes the following steps:

[0046] S501. Calculate the first vector from the wrist to the finger root;

[0047] S502. Calculate the second vector from the finger root to the fingertip;

[0048] S503. Calculate the angle between the first vector and the second vector.

[0049] Further, step 6 includes the following steps:

[0050] S601. Set the thumb angle threshold to 53°, the first angle threshold to 65°, and the second angle threshold to 49°;

[0051] S602. If the angle of the thumb is greater than the thumb angle threshold, go to step S603, otherwise go to step S605;

[0052] S603. If the angles of the other four fingers are greater than the first angle threshold, it is determined as a fist gesture, otherwise go to step S604;

[0053] S604. Determine as 1, 2, 3, 4 according to the number of fingers whose angles of the other four fingers are less than the second angle threshold;

[0054] If the angle of the thumb is less than the second angle threshold and the angles of the other four fingers are greater than the first angle threshold, it is determined as the thumb gesture; otherwise, proceed to step S606;

[0055] S606: If the angles of the thumb and the little finger are less than the second angle threshold and the angles of the other three fingers are greater than the first angle threshold, it is determined as the number 6; otherwise, proceed to step S607;

[0056] S607: If the angles of the thumb and the index finger are less than the second angle threshold and the angles of the other three fingers are greater than the first angle threshold, it is determined as the number 8; otherwise, proceed to step S608;

[0057] S608: If the angles of the thumb, index finger, middle finger, ring finger, and little finger are all less than the second angle threshold, it is determined as an open palm.

[0058] Furthermore, in order to reduce the model complexity while ensuring the model accuracy, both the palm detection model and the hand key point detection model of the present invention adopt the DHRNet network model.

[0059] According to the present invention of the above solution, its beneficial effects are as follows:

[0060] By using the image output optical machine and the waveguide diffraction lens, the present invention can provide a clearer image. The waveguide diffraction lens of the double diffraction optical machine component can achieve a thinner and lighter design, and helps to naturally superimpose virtual information onto the user's field of vision without causing obvious visual distortion or discomfort;

[0061] The built-in gesture recognition component enables users to send commands through simple gestures for gesture communication; the combined use of the equipped camera and the TOF sensor can not only enable the device to better understand the surrounding environment, but also achieve functions such as accurate spatial positioning and object distance measurement;

[0062] The backplane of the device host is directly connected to the front lower edge of the headband wearing member, which helps to improve the comfort during long-term wearing and ensures a good user experience even in a continuous working state. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a schematic structural diagram of the present invention;

[0064] Figure 2 is a schematic structural diagram of another perspective of the present invention;

[0065] Figure 3 is an exploded view of the structure of the present invention;

[0066] Figure 4 is a schematic structural diagram of the headband wearing member in the present invention;

[0067] Figure 5 Schematic diagram of the back structure of the device host in the present invention;

[0068] Figure 6 Exploded view of the structure of the device host of the present invention;

[0069] Figure 7 Schematic diagram of the inclined arrangement of the waveguide diffraction lens in the present invention;

[0070] Figure 8 Exploded view of the structure of the double diffraction optical machine component in the present invention;

[0071] Figure 9 Exploded view of the structure of the double diffraction optical machine component from another perspective;

[0072] Figure 10 Schematic diagram of the structure of the thermal imaging lens assembly in a preferred embodiment;

[0073] Figure 11 Schematic diagram of the structure of the existing HRNet network model;

[0074] Figure 12 Schematic diagram of the structure of the DHRNet network model of the present invention.

[0075] In the figure,

[0076] 1. Headband wearing member; 101. Front part; 102. Leg; 11. Rotating shaft; 12. Electrical socket; 13. Concave arc structure; 14. First wire passing hole;

[0077] 2. Device host; 20. Placing slot; 21. Accommodating cavity; 211. Cavity cover plate; 22. Shaft groove; 221. Axle pin; 23. Arc-shaped slot; 24. Second wire passing hole;

[0078] 3. Double diffraction optical machine component; 31. Image output optical machine; 32. Waveguide diffraction lens; 321. Arc-shaped notch; 33. Main optical machine housing; 331. Square heat dissipation hole; 332. Circular light outlet hole; 333. Installation groove; 34. Side housing cover plate; 35. Housing back plate; 351. Limiting groove; 352. Convex arc surface; 353. Positioning pin;

[0079] 41. Camera; 42. Infrared LED fill light; 43. TOF sensor; 44. Gesture recognition lens assembly;

[0080] 5. Protective goggles; 51. Clamping groove; 52. Supporting groove;

[0081] 6. Thermal imaging lens assembly; 61. Manual focusing ring; 62. Lens holder; 63. Plug connector;

[0082] 7. Elastic band;

[0083] 8. First power supply host; 81. Through-hole; 82. Concave arc surface;

[0084] 9. Second power supply host; 91. Data line. Detailed implementation manner

[0085] In order to better understand the purpose, technical solution and technical effect of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and embodiments. It should be noted that: similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, it is stated that the embodiments described below are only used to explain the present invention and are not used to limit the present invention.

[0086] It should be noted that when an element is referred to as "fixed to" or "disposed on" another element, it can be directly on the other element or there can also be an intermediate element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intermediate element at the same time.

[0087] The indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this application is usually placed during use, or the orientation or positional relationship commonly understood by those skilled in the art, or the orientation or positional relationship in which the product of this application is usually placed during use. It is only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to this application.

[0088] The terms "first" and "second" are only used for the purpose of convenient description, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of technical features.

[0089] As Figures 1 to 3 shown, a multifunctional detection, recognition and interaction system based on an AR glasses includes a headband wearing member 1 and a device host 2. The headband wearing member 1 and the device host 2 can be fixedly connected, rotatably connected, or detachably connected. By wearing the headband wearing member 1, the user realizes fixing the device host 2 on the forehead of the user.

[0090] In this embodiment, the back plate of the device host 2 is connected to the lower edge of the front portion 101 of the headband wearing part 1, so that the headband wearing part 1 can avoid the heat dissipation holes on the back of the device, and prevent the heat dissipation effect of the device host 2 from being affected by the obstruction. The device host 2 is connected to the lower edge of the headband wearing part 1, and the upper surface of the device host 2 can be located below the front portion 101 of the headband wearing part 1, so that the upper surface of the device host 2 forms a placement slot 20 in front of the headband wearing part 1. When the user wears a safety helmet, the brim of the safety helmet can be placed on the placement slot 20, the AR glasses can be used with the safety helmet, and the device host 2 can bear part of the weight of the safety helmet; when the user takes off the safety helmet, there is no need to take off the AR glasses. It can be seen that the use of AR glasses is independent of the safety helmet and can be firmly worn on the user's head.

[0091] The device host 2 is in the shape of a crescent, and the width of the device host 2 is equal to the distance between the two side legs 102 of the headband wearing piece 1, so that the size of the device host 2 is closer to the visor, increasing the contact area with the visor, which is conducive to the stability of wearing AR glasses and a helmet at the same time. The working mainboard inside the device host 2 is also in the shape of a crescent to adapt to the internal structure of the device host 2, and at the same time has sufficient space for the layout of electronic components, providing a physical basis for the realization of multifunctional AR glasses.

[0092] The top of the front portion 101 of the headband wearing piece 1 is tilted backwards, so that the front portion 101 of the headband wearing piece 1 can be smoothly inserted into the gap between the safety helmet and the wearer's head, and the back side of the front portion 101 of the headband wearing piece 1 can fit the forehead of the user (or wearer), thereby increasing the wearing contact area, reducing the gravity pressure of the device and the safety helmet, and improving wearing comfort.

[0093] The front part 101 and the two side legs 102 of the headband wearing piece 1 are integrally formed to form a ring with an opening, and the two side legs 102 have a certain elasticity, which is convenient for the user to open the two legs 102 and wear. An elastic band 7 is provided at the opening of the headband wearing piece 1, and the two ends of the elastic band 7 are respectively connected to the two side legs 102, and the elastic band 7 is used to adjust the tightness of the headband wearing piece 1.

[0094] like Figure 5 As shown, in a preferred embodiment, a rotating shaft 11 is provided at the lower edge of the front portion 101 of the headband wearing part 1, an axis groove 22 is provided on the back of the device main body 2, an axis pin 221 is provided in the axis groove 22, and the rotating shaft 11 is rotatably connected with the axis pin, so that the device main body 2 can rotate along the rotating shaft 11 relative to the headband wearing part 1, and the rotation angle is 0 to 60°.

[0095] like Figure 4As shown, in an alternative embodiment, the junction of the front portion 101 of the headband wearing member 1 and the support leg 102 has a concave arc structure 13. The concave arc structure 13 can reduce the weight of the front portion 101 of the headband wearing member 1, which is beneficial to reducing the overall front-end weight of the AR glasses product and avoiding the excessive front-end weight of the product from affecting the wearing stability and comfort. The back side of the headband wearing member 1 (the side in contact with the user's head) is provided with a flexible material such as silica gel or sponge, which can prevent the hard material shell from damaging the user's head. Therefore, the flexible back side of the headband wearing member 1 can improve the wearing comfort and safety.

[0096] In the present invention, an electrical socket 12 is provided at the end of one of the support legs 102 of the headband wearing member 1. The electrical socket 12 is electrically connected to the working main board of the device host 2; the first power supply host 8 is plugged into the electrical socket 12 using a power cord to supply electrical energy to the working main board. Among them, a flexible circuit board is provided inside the headband wearing member 1, and a wiring channel is provided at the connection between the headband wearing member 1 and the device host 2. One end of the flexible circuit board is connected to the electrical socket 12, and the other end is electrically connected to the working main board through a connecting wire in the wiring channel.

[0097] The output interface of the first power supply host 8 is arranged on one side close to the electrical socket 12, which is convenient for connecting the output interface of the first power supply host 8 and the electrical socket 12 of the headband wearing member 1 on the same side, and is beneficial to shortening the length of the connecting wire.

[0098] In a specific embodiment, the front housing of the headband wearing member 1 is provided with a first wire passing hole 14, and the housing back plate 35 of the device host 2 is correspondingly provided with a second wire passing hole 24. The first wire passing hole 14 serves as one end of the wiring channel between the headband wearing member 1 and the device host 2, and the second wire passing hole 24 serves as the other end of the wiring channel. And a connecting wire for connecting the flexible circuit board and the working main board is arranged inside the wiring channel. In an AR glasses product in which the headband wearing member 1 and the device host 2 are rotatably connected structures, the first wire passing hole 14 is arranged on the front housing of the headband wearing member 1 close to one side of the rotating shaft 11, the second wire passing hole 24 is arranged on the side wall of the shaft groove 22, and the first wire passing hole 14 and the second wire passing hole 24 are located on the same side of the rotating shaft 11.

[0099] The two ends of the first power supply host 8 are provided with through holes 81. The elastic band 7 is detachably connected to the through holes 81 to mount the first power supply host 8 on the elastic band 7. Specifically, one end of the elastic band 7 is connected to the end of the left support leg 102, and the other end sequentially passes through the left through hole 81 of the first power supply host 8, the right through hole 81 of the first power supply host 8, and then is connected to the end of the right support leg 102. The bottom surface of the first power supply host 8 (the side close to the user's head) is a concave arc surface 82. The concave arc surface structure design is beneficial to better fitting the back of the user's head and improving the wearing comfort.

[0100] In the present invention, a dual-diffraction optical machine assembly 3 is provided in the middle of the bottom of the device host 2. The dual-diffraction optical machine assembly 3 includes an image output optical machine 31 and a waveguide diffraction lens 32. The image output optical machine 31 is used to output an image source to the waveguide diffraction lens 32, and the waveguide diffraction lens 32 is used to image the image source within the left and right eye visual field regions of the wearer, achieving the effect of left and right dual-channel augmented reality.

[0101] In an application example, the image source of the image output optical machine 31 comes from the cloud or a background terminal. Specifically, in the fire protection industry, the command center monitors the on-site environment through a camera 41, analyzes the on-site fire situation and formulates a rescue plan in the background terminal, and then remotely sends data such as commands or analysis information to the AR glasses of on-site firefighters. Finally, it is imaged to the waveguide diffraction lens 32 through the dual-diffraction optical machine assembly 3, enabling firefighters to obtain information synchronously and communicate with the command center in real time.

[0102] In other application examples, the image source of the image output optical machine 31 comes from the storage unit within the device host 2. The working mainboard of the device host 2 is at least integrated with a processing unit (MCU) and a storage unit. The processing unit can analyze the environment based on the real scene and spatial positioning data obtained by the camera 41 and / or sensors of the AR glasses, and automatically calculate how to accurately place virtual content at the corresponding position in the real world, retrieve the display content from the storage unit and display it at the corresponding position. For example, if it is analyzed that the position of the cabinet in front is 2 meters away from the user, the text information "2 meters" and the distance scale icon are correctly displayed on the cabinet.

[0103] Such as Figure 8 and Figure 9As shown, in an alternative embodiment, the dual-diffraction optical engine assembly 3 not only includes an image output optical engine 31 and a waveguide diffraction lens 32, but also includes: an optical engine main housing 33, a housing side cover plate 34, and a housing back plate 35. The optical engine main housing 33 has a chamber for mounting the image output optical engine 31, and the chamber has a lateral opening. The housing side cover plate 34 is detachably mounted at the lateral opening of the optical engine main housing 33, and the optical engine main housing 33 and the housing side cover plate 34 are assembled into a protective housing. One side of the optical engine main housing 33 is provided with a square heat dissipation hole 331, and the opposite side is provided with a circular light outlet hole 332, and the square heat dissipation hole 331 and the circular light outlet hole 332 are located on both sides of the lateral opening. The square heat dissipation hole 331 can provide a heat dissipation outlet for the image output optical engine 31 inside the protective housing, facilitating the discharge of the heat generated by the operation of the image output optical engine 31. The size of the circular light outlet hole 332 is equal to the size of the light port of the image output optical engine 31, which can provide a larger light outlet for the image output optical engine 31 inside the protective housing, and can meet the requirement that the image output optical engine 31 uses a shorter focal length lens to achieve the same field of view angle, facilitating the projection of the image onto the waveguide diffraction lens 32 at a short distance; moreover, the larger light outlet can transmit more light, which helps to improve the overall brightness of the picture, thereby providing a clear virtual image.

[0104] The protective housing assembled by the optical engine main housing 33 and the housing side cover is provided with a mounting groove 333 for mounting the waveguide diffraction lens 32 on the side close to the circular light outlet hole 332. The housing back plate 35 is correspondingly provided with a limiting groove 351, and the limiting groove 351 is adapted to the structure of the waveguide diffraction lens 32. The housing back plate 35 can be detachably mounted on the protective housing, and the waveguide diffraction lens 32 is fixed by the mutually spliced limiting groove 351 and mounting groove 333. Screw holes are provided at the upper and lower ends of the housing back plate 35. When maintaining or replacing the waveguide diffraction lens 32, the housing back plate 35 can be disassembled by screwing the screws, and then the waveguide diffraction lens 32 can be removed. When installing the waveguide diffraction lens 32, the waveguide diffraction lens 32 is fixed by the combined structure of the limiting groove 351 and the mounting groove 333, without complicated lens debugging, which is very convenient.

[0105] Positioning pins 353 are provided on both sides of the upper and lower screw holes of the housing back plate 35, and corresponding pin holes are provided on the back of the optical engine main housing 33. When installing the housing back plate 35, it only needs to align the positioning pins 353 and insert them into the pin holes to quickly complete the installation.

[0106] An arc-shaped notch 321 is provided on the lower side of the middle part of the waveguide diffraction lens 32. The width of the middle lens only needs to satisfy covering the circular light-emitting hole 332 so that light can be smoothly coupled into the waveguide diffraction lens 32. The arc-shaped notch 321 can not only play a role in installation and positioning, but also reduce the lens material to save costs. Correspondingly, a convex arc surface 352 is provided at the lower end of the limiting groove 351 of the housing back plate 35, and the convex arc surface 352 fits with a part of the lower surface of the arc-shaped notch 321 of the waveguide diffraction lens 32 to ensure the stable installation of the waveguide diffraction lens 32.

[0107] As Figure 7 shown, in a preferred embodiment, the waveguide diffraction lens 32 is inclined. Specifically, its top end is close to the image output optical machine 31, and the bottom end is far from the image output optical machine 31; the inclination angle α of the waveguide diffraction lens 32 ranges from 10° to 15°. The inclined waveguide diffraction lens 32 can not only make the light of the image output optical machine 31 better coupled into the waveguide diffraction lens 32, but also prevent the ambient light below the AR glasses from being reflected to the human eye through the inner mirror surface of the waveguide diffraction lens 32, affecting the viewing of the image on the waveguide diffraction lens 32. Correspondingly, the installation groove 333 of the optical machine main housing 33 and the limiting groove 351 of the housing back plate 35 are both inclined to adapt to the inclined state of the waveguide diffraction lens 32.

[0108] As Figure 5 and Figure 6 shown, in an alternative embodiment, a receiving cavity 21 for installing the double diffraction optical machine assembly 3 is formed by extending downward in the middle of the bottom of the housing of the device host 2, and the inner cavity of the device host 2 is communicated with the receiving cavity 21, which is convenient for the wiring row of the double diffraction optical machine assembly 3 to pass through the receiving cavity 21 into the inner cavity of the device host 2 and be electrically connected to the working main board. The receiving cavity 21 is provided with a detachable cavity cover plate 211. Two stud bolts are provided inside the receiving cavity 21, and the cavity cover plate 211 is correspondingly provided with bolt holes. The optical machine main housing 33 of the double diffraction optical machine assembly 3 is provided with connecting ears with screw holes. The double diffraction optical machine assembly 3 is placed into the receiving cavity 21, the cavity cover plate 211 is covered, and the screw holes of the stud bolts, the screw holes of the connecting ears, and the bolt holes of the cavity cover plate 211 are correspondingly aligned. The bolt passes through from the outside to the inside, and tightening the bolt can fix and install the double diffraction optical machine assembly 3 in the receiving cavity 21. Notches for avoiding the waveguide diffraction lens 32 are provided on the side walls of the receiving cavity 21 and the side walls of the cavity cover plate 211.

[0109] In the present invention, a transparent protective lens 5 is provided on the front side of the bottom of the device host 2. The material of the protective lens 5 can be polycarbonate. The polycarbonate protective lens 5 has the characteristics of impact resistance and light weight; the material of the protective lens 5 can also be glass. The glass protective lens 5 has a higher hardness, a harder and smoother surface, and is not easily scratched.

[0110] The bottom of the protective goggles 5 is provided with a supporting groove 52. When the user wears the AR glasses, the user's nose bridge is placed into the supporting groove 52, increasing the contact area between the nose bridge and the protective goggles 5, reducing the gravitational pressure of the protective goggles 5 on the nose bridge, and improving the wearing comfort. Preferably, a flexible material such as silica gel or sponge can be provided at the bottom of the supporting groove 52.

[0111] The protective goggles 5 are arc-shaped. The chord height of the protective goggles 5 is not less than 2 cm, and the width of the protective goggles 5 is greater than the width of the waveguide diffraction lens 32 to provide sufficient space for installing the double diffraction optical machine assembly 3 and avoid affecting the wearing; at the same time, the protective goggles 5 form a semi-surrounding protection structure in front of the double diffraction optical machine assembly 3.

[0112] As Figure 5 shown, an arc-shaped slot 23 is provided at the bottom of the housing of the device host 2 corresponding to the position of the protective goggles 5. A clamping block is provided on the inner wall of the housing adjacent to the arc-shaped slot 23. A clamping groove 51 is provided at the top of the protective goggles 5. The number and position of the clamping groove 51 and the clamping block correspond one by one. Through the above structure, the detachable installation of the protective goggles 5 and the device host 2 can be realized, which is convenient for maintenance personnel to repair or replace the aging protective goggles 5.

[0113] As Figure 1 shown, in the present invention, a camera 41 is provided in the middle of the front side of the device host 2. The camera 41 is used to capture the real scene images of the surrounding environment, and through calculation and processing by the processing unit, the spatial positioning and three-dimensional mapping of the surrounding environment are realized. Based on the real-world images captured by the camera 41, the AR glasses can superimpose virtual information or graphics thereon in real time. The shooting data obtained by the camera 41 can be stored in the storage unit, and can also be transmitted to other devices or uploaded to the cloud through a wireless network or a mobile network.

[0114] Infrared LED fill lights 42 are provided on both sides of the camera 41. In a completely dark environment, the infrared light emitted by the infrared LED fill lights 42 irradiates an object, and the CCD or CMOS sensor in the camera 41 captures the reflected infrared light and converts it into an image, realizing the infrared night vision function of the camera 41.

[0115] A TOF sensor 43 is also provided in the middle of the front side of the device host 2. The TOF sensor 43 is located below the camera 41; the TOF sensor 43 is used to measure the distance between an object and the TOF sensor 43 in real time; the TOF sensor 43 can measure a distance value for each point in the field of view, thereby generating a complete depth map. The depth map shows the depth information of each position, that is, the actual distance corresponding to each pixel point. The generated depth map will be further processed, and the AR glasses system reconstructs the three-dimensional environment around the user based on the depth map and places virtual objects or performs other operations accordingly.

[0116] On the side of the device host 2, there is a thermal imaging lens assembly 6. The thermal imaging lens assembly 6 generates images by detecting the infrared radiation (heat) emitted by objects, and can display the temperature distribution of different objects; through the temperature information captured by the thermal imaging lens, the AR glasses system can superimpose virtual information, such as temperature readings, alarm prompts, etc., in the user's field of vision.

[0117] In a specific application example, AR glasses with thermal imaging function play a very important role in the field of fire fighting. The thermal imaging lens can quickly detect and display high-temperature areas, helping firefighters quickly locate the position of the fire source. In some cases, the fire source may be blocked by walls, furniture or other obstacles. The thermal imaging lens can detect the heat penetrating the obstacles to help firefighters discover hidden fire sources, which is conducive to firefighters taking measures earlier; through the thermal imaging image, firefighters can intuitively see the spread direction and speed of the fire, so as to better plan the fire extinguishing strategy and evacuation route; in an environment with thick smoke and low visibility, by using the obvious difference between the heat emitted by the human body and the surrounding environment, the thermal imaging lens helps firefighters find trapped people, making the rescue work more efficient; after the fire is extinguished, the thermal imaging lens can be used to detect the positions where there may be embers or overheating points to prevent re-ignition, etc.

[0118] As Figure 10 shown, in a preferred embodiment, the thermal imaging lens assembly 6 is provided with a manual focus ring 61. By rotating the manual focus ring 61, the focal length of the thermal imaging lens can be adjusted to obtain the required clarity. The clarity of the thermal imaging map directly affects the user's ability to identify and analyze details. Taking the aspect of detail recognition as an example, a clear image can help the user more accurately identify and distinguish different objects or regions. For example, in industrial inspection, a clear image can more easily detect hot spots or defects in equipment. In addition, since the autofocus system may be affected in some extreme environments (such as high temperature, high humidity, low light, etc.), the thermal imaging lens with manual focus function is more reliable in the high-temperature fire scene.

[0119] In an alternative embodiment, the thermal imaging lens assembly 6 has a lens mount 62. One side of the lens mount 62 is provided with a plug connector 63, and the device host 2 is correspondingly provided with a plug slot. Through the connection structure of the plug connector 63 and the plug slot, the thermal imaging lens assembly 6 is detachably connected to the device host 2 and powered by the device host 2. This embodiment meets the user's requirement to select and match the AR glasses product according to the actual situation, with higher flexibility; it is also conducive to reducing the maintenance cost. When the thermal imaging lens fails or is damaged, the thermal imaging lens assembly 6 can be simply replaced instead of the whole device.

[0120] In the present invention, the working motherboard of the device host 2 is integrated with a Beidou module and / or a GPS module, and an inertial measurement module. The Beidou module and / or the GPS module obtains the accurate current position information of the user by receiving Beidou or GPS satellite signals. The inertial measurement module is used to detect the motion state of the user, including the walking direction, speed, posture, etc.

[0121] The AR glasses system accesses the map database in the cloud or the storage unit, calculates the optimal path based on the user's current position and destination, and generates navigation instructions. The AR glasses system displays the navigation instructions, such as navigation arrows, distance prompts, etc., through the waveguide diffraction lens 32, and finally realizes the superimposition of navigation information in the user's field of view.

[0122] In the present invention, the AR glasses are configured with a second power supply host 9. The optional size of the second power supply host 9 is 115×70×34mm, which is convenient for the operator to place the second power supply host 9 in the arm pocket or the chest pocket. The second power supply host 9 is connected to the electrical socket 12 of the device host 2 through a data cable 91.

[0123] In an optional embodiment, the first power supply host 8 is a control host, and the second power supply host 9 is a mobile charging power supply.

[0124] In another optional embodiment, the first power supply host 8 is a mobile charging power supply, and the second power supply host 9 is a control host.

[0125] Among them, the mobile charging power supply is connected to the device host 2 through the electrical socket 12 of the headband wearing member 1 to provide the electrical energy required for the working motherboard. The control host is connected to the device host 2 through the electrical socket 12 of the headband wearing member 1, which can not only supply power to the device host, but also send and receive data with the device host 2, and send control instructions to the device host 2. Specifically, the control host is provided with a microphone and stereo headphones. The main control board of the control host is integrated with a processor, a cellular network module (4G / 5G), an image codec module, and an audio codec module. The processor is used to coordinate the work of each module to process video and audio data. The control host receives the audio and video data of the other party through the 4G / 5G network; the audio codec module decodes the received audio data stream and converts it into an audible audio signal, and the stereo headphones play the audio; the image codec module decodes the received video data stream and converts it into a displayable image frame, and the imaging output optical machine 31 outputs the image frame.

[0126] The camera 41 of the device host 2 captures the video image in front of the user, encodes the video data stream through the processing unit, and transmits it to the control host; the control host collects the user's voice through the microphone and encodes it into an audio data stream; the control host sends the audio and video data to the other party through the 4G / 5G network. It can be seen that the present invention configures the control host to add the voice and video call function of the AR glasses. Through network connection, similar to the live broadcast method, the image in front of the eyes is sent to the other party of the call, and real-time communication can be carried out.

[0127] In an alternative embodiment, the control host is provided with a display screen or a touch screen, and several buttons are provided below the screen. The display content of the display screen includes but is not limited to: network operator, network connection status, battery power, date, time, and text messages. The buttons include four direction keys of up, down, left, and right and an enter key, which are convenient for setting operations, selection operations, page turning operations, etc. on the control host. When the screen is a touch screen, the user can perform the above operations through the touch screen.

[0128] As Figure 2 shown, in the present invention, a gesture recognition lens assembly 44 is provided at the bottom of the device host 2 near the right side of the human hand, and the shooting direction of the gesture recognition lens assembly 44 is downward. It can be seen that the gesture recognition area of the AR glasses in this solution is on the lower right side of the AR glasses, and the right hand gesture is photographed at close range. This design conforms to the right hand usage habits of most people.

[0129] The present invention also provides a usage method of the multifunctional detection and recognition interaction system based on the AR glasses of the above solution:

[0130] Step A: Fix the headband wearing member on the head to ensure that the device host is in front of the forehead and the double diffraction optical machine assembly is in front of the bridge of the nose, and the waveguide diffraction lens is facing both eyes;

[0131] Step B: Start the control host and the device host, and the system performs self-check and initialization, including the camera, infrared LED fill light, TOF sensor, gesture recognition component, and thermal imaging lens;

[0132] Step C: The camera captures the real scene image of the surrounding environment, and the TOF sensor measures the distance between the object and the sensor in real time, and the system analyzes the user's environment;

[0133] Step D: Automatically calculate the position of the virtual content according to the data, retrieve the corresponding content, and superimpose the virtual image on the user's real vision through the waveguide diffraction lens;

[0134] Step E: Adjust the focal length of the thermal imaging lens through the manual focus ring;

[0135] Step F: The thermal imaging lens assembly captures the thermal radiation of the object and converts it into a visualized thermal image;

[0136] Step G: The gesture recognition component captures and recognizes the user's gesture, and transmits the recognized gesture signal to the processing unit to output corresponding instructions or corresponding information;

[0137] Step H: The control host communicates remotely with other control hosts or the backend platform through the 4G / 5G network, including sending text messages and real-time audio and video broadcasts.

[0138] In step G, the gesture recognition component captures and recognizes the user's gesture, including:

[0139] S1: Read in the pre-trained palm detection model and hand key point detection model;

[0140] S2: The gesture recognition lens captures the hand image;

[0141] S3: Use the palm detection model to locate the palm area in the hand image;

[0142] S4: Use the hand key point detection model to obtain the key point positions of the hand;

[0143] S5: Use the position information of the key points to calculate the finger angles through an algorithm;

[0144] S6: Compare the calculated finger angles with the preset threshold values to determine the gesture type;

[0145] Among them, the calculation of the finger angles in step S5 includes the following steps:

[0146] S501: Calculate the first vector from the wrist to the finger root;

[0147] S502: Calculate the second vector from the finger root to the fingertip;

[0148] S503: Calculate the angle between the first vector and the second vector.

[0149] Among them, step S6 includes the following steps:

[0150] S601: Set the thumb angle threshold to 53°, the first angle threshold to 65°, and the second angle threshold to 49°;

[0151] S602: If the angle of the thumb is greater than the thumb angle threshold, go to step S603, otherwise go to step S605;

[0152] S603: If the angles of the other four fingers are greater than the first angle threshold, it is determined as a fist gesture, otherwise go to step S604;

[0153] S604: Determine as 1, 2, 3, 4 according to the number of fingers whose angles of the other four fingers are less than the second angle threshold;

[0154] S605. If the angle of the thumb is less than the second angle threshold and the angles of the other four fingers are greater than the first angle threshold, it is determined as the thumb gesture; otherwise, go to step S606;

[0155] S606. If the angles of the thumb and the little finger are less than the second angle threshold and the angles of the other three fingers are greater than the first angle threshold, it is determined as the number 6; otherwise, go to step S607;

[0156] S607. If the angles of the thumb and the index finger are less than the second angle threshold and the angles of the other three fingers are greater than the first angle threshold, it is determined as the number 8; otherwise, go to step S608;

[0157] S608. If the angles of the thumb, index finger, middle finger, ring finger, and little finger are all less than the second angle threshold, it is determined as an open palm.

[0158] In order to reduce the model complexity while ensuring the model accuracy, both the palm detection model and the hand key point detection model of the present invention adopt the DHRNet network model, which is improved on the existing HRNet network model.

[0159] As Figure 11 shown, first, the HRNet network model first performs two convolutional operations on the input image for downsampling, reducing the resolution of the input image by 4 times to reach 1 / 4 of the initial size, and then enters the main network. The main part of the HRNet network consists of four stages. Among them, stage 1 is composed of 4 Bottleneck residual units. After stage 1, there are a series of exchange structures and stage structures. Each time passing through an exchange structure, the network will add a scale branch through the downsampling operation. In addition, in order to obtain multi-scale information, each stage in the network will fuse the features of different scales. Stages 2, 3, and 4 are respectively stacked 1, 4, and 3 times. Among them, the last exchange unit in stage 4 only retains the feature layer with the highest resolution, and finally, hand point prediction is performed on the high-resolution feature map.

[0160] Observing the HRNet network structure, it can be found that as the number of low-resolution sub-networks increases, the network branches and convolutional modules also increase accordingly, resulting in an exponential increase in the number of network parameters and the computational overhead of the model as the order increases. However, in practical applications, the requirements of high speed and high accuracy are often required.

[0161] As Figure 12As shown, on the basis of retaining the first three stages (Stage 1, Stage 2, and Stage 3) of the DHRNet network model, Stage 4 is changed to Stage D. Stage D fuses multi-scale features through transposed convolution, significantly reducing the number of model parameters. Additionally, a transposed convolution is added again to the highest-resolution feature map of its output to generate a higher-resolution feature map for predicting the heatmap.

[0162] To better handle scale changes and enhance the network's perception ability of learning errors, a multi-resolution supervision strategy is adopted during training. Specifically, the loss functions for predicting the heatmap are calculated separately for the highest-resolution (1 / 4 resolution of the input) branch and the higher-resolution (1 / 2 resolution of the input) branch after transposed convolution in Stage D. The final loss value of the heatmap is the sum of the losses of the above two branches.

[0163] In Stage D, the 1 / 4-resolution heatmap containing i key points and the finally obtained 1 / 2-resolution heatmap after transposed convolution operation are the learning targets of the network. A two-dimensional Gaussian distribution function is used to generate the true heatmap h i (x,y):

[0164]

[0165] where (x,y) represents the position coordinates of any pixel point in the true heatmap, (x i ,y i ) is the center of the Gaussian function, representing the true position coordinates of the i-th key point, and represents the preset standard deviation of the Gaussian function for each key point.

[0166] The predicted heatmap is calculated by the following formula:

[0167]

[0168] The loss function is divided into classification loss and regression loss, and the mean square error MSE loss function is used for calculation. The formula is as follows:

[0169]

[0170] where Nb represents the number of hand key points, and represent the predicted heatmap and the true heatmap of the b-th heatmap respectively.

[0171] Denote the loss value of the highest-resolution (1 / 4 resolution of the input) branch in Stage D by , and denote the loss value of the higher-resolution (1 / 2 resolution of the input) branch after transposed convolution by . The total loss value L of the final key point detection heatmap is:

[0172]

[0173] Finally, experiments are provided: The improved DHRNet network is experimentally verified on the COCO dataset, as follows:

[0174] 1. For the COCO keypoint detection dataset, the Object Keypoint Similarity (OKS) metric is designed to represent the similarity between the ground truth and the predicted hand keypoints. The OKS calculation method is as follows:

[0175]

[0176] where i represents the i-th hand keypoint, d i represents the Euclidean distance between the predicted position and the ground truth position of the i-th hand keypoint, s represents the scale of the object, is the area of the object detection box, k i represents the normalization factor of the hand point, which is a constant used to control attenuation. v i represents the visibility of the hand point, which can take values of 0, 1, or 2. 0 means the hand point is not labeled, 1 means the hand point is labeled but occluded and invisible, and 2 means the hand point is labeled and visible. represents taking the value of 1 when and 0 otherwise.

[0177] 2. Average Precision AP:

[0178]

[0179] where p represents the number of objects, T is a pre-set OKS threshold, and only when the OKS value is greater than the threshold does it indicate a correct prediction. is used to represent the relationship between OKS and the threshold T, and takes the value of 1 if OKS is greater than T.

[0180] represents the accuracy rate when T = n, represents the average accuracy rate of medium-scale objects, represents the average accuracy rate of large-scale objects, represents the average recall rate.

[0181] First, adjust the aspect ratio of the height to the width of the hand detection box to 4:3, then crop the box from the image and resize it to a fixed input size. To reduce the computational cost and experimental cycle, the input size of the experimental network is set to 256 pixels x 192 pixels. In addition, the data augmentation method adopts random rotation, random scale transformation, and flipping. The random rotation range is [-45°, 45°], and the random scale transformation range is [0.65, 1.35]. It should be noted that 0.65 and 1.35 have no units, and they represent the scale factors relative to the original image size.

[0182]

[0183] The experimental results show that improving Stage 4 to Stage D can effectively reduce the number of network parameters and computational amount, and the accuracy is basically the same as before.

[0184] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described.

[0185] The above embodiments only represent several implementation manners of the present invention. The description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the invention patent should be subject to the appended claims.

Claims

1. A multifunctional detection and recognition interactive system based on AR glasses, characterized in that: The device comprises a headband wearing piece and a main device, wherein the back plate of the main device is connected to the front lower edge of the headband wearing piece, the front lower edge of the headband wearing piece is provided with a rotating shaft, the back of the main device is provided with an axis groove, the axis groove is provided with an axis pin, the rotating shaft is rotatably connected with the axis pin, and the main device can rotate relative to the headband wearing piece along the rotating shaft, and the rotation angle is 0 to 60°; A dual-diffraction optical machine component and a gesture recognition component are provided at the bottom of the device host. The dual-diffraction optical machine component includes an image output optical machine, a waveguide diffraction lens, an optical machine main shell, a shell side cover plate and a shell back plate. The image output optical machine is used to output an image source to the waveguide diffraction lens, and the waveguide diffraction lens is used to image the image source into the left and right eye fields of the wearer; the waveguide diffraction lens is tilted, with the top of the waveguide diffraction lens close to the image output optical machine and the bottom away from the image output optical machine; the tilt angle α of the waveguide diffraction lens ranges from 10° to 15°; The optical machine main housing has a cavity for installing the image output optical machine, the cavity has a lateral opening, the housing side cover can be detachably installed at the lateral opening of the optical machine main housing to be assembled with the optical machine main housing into a protective housing, one side of the optical machine main housing is provided with a square heat dissipation hole, and the other side opposite is provided with a circular light exit hole, the square heat dissipation hole and the circular light exit hole are located on both sides of the lateral opening; the protective housing is provided with a mounting groove on the side close to the circular light exit hole, and the housing back plate is correspondingly provided with a limiting groove, the housing back plate can be installed and removed on the protective housing, and the waveguide diffraction lens is fixed by the combined structure of the limiting groove and the mounting groove; The gesture recognition component is used to capture and recognize the wearer's gestures; A camera and a TOF sensor are provided in the middle of the front side of the device host, the camera is used to capture the real-life image of the surrounding environment, and the TOF sensor is used to measure the distance between the object and the TOF sensor in real time; The gesture recognition component captures and recognizes the wearer's gesture, including: S1, read in the pre-trained palm detection model and hand key point detection model; S2, hand image captured by the gesture recognition camera; S3, using a palm detection model to locate the palm area in the hand image; S4, using the hand key point detection model to obtain the key point position of the hand; S5, using the position information of the key points, calculate the finger angle through an algorithm; S501, calculating a first vector from the wrist to the base of the finger; S502, calculating a second vector from the base of the finger to the fingertip; S503, calculating the angle between the first vector and the second vector; S6, judging the gesture type by comparing the calculated finger angle with a preset threshold; The palm detection model and the hand key point detection model both adopt the DHRNet network model. The first three stages of the DHRNet network model adopt the first three stages of the HRNet network model, and the fourth stage of the DHRNet network model calculates the loss function of the predicted heat map by the highest resolution branch and the higher resolution branch after transposed convolution respectively. The loss value of the final heat map is the sum of the losses of the above two branches, where the highest resolution is 1 / 4 of the input resolution and the higher resolution is 1 / 2 of the input resolution. Specifically, The 1 / 4 resolution heat map containing i key points and the 1 / 2 resolution heat map obtained after the transposed convolution operation are used as the learning target of the network. The two-dimensional Gaussian distribution function is used to generate the real heat map hi (x, y), and the predicted heat map hi* (x, y) is obtained using the following formula: ； The total loss value L of the key point detection heat map is: ； Among them, Loss1 / 4 represents the input 1 / 4 resolution branch loss value, and Loss1 / 2 represents the 1 / 2 resolution branch loss value after transposed convolution.

2. The multifunctional detection and recognition interactive system based on AR glasses according to claim 1 is characterized in that: A thermal imaging lens assembly is provided on the side of the device host, and the thermal imaging lens assembly is provided with a manual focus ring, and the focal length of the thermal imaging lens is adjusted by rotating the manual focus ring.

3. The multifunctional detection and recognition interactive system based on AR glasses according to claim 1 is characterized in that: The working mainboard of the device host is integrated with a positioning module and an inertial measurement module. The positioning is used to obtain the current position of the user, and the inertial measurement module is used to detect the movement state of the user.

4. The multifunctional detection and recognition interactive system based on AR glasses according to claim 1 is characterized in that: It also includes a control host, which is electrically connected to the device host through a data line to transmit data, or sends a control instruction to the device host.

5. The multifunctional detection and recognition interactive system based on AR glasses according to claim 4 is characterized in that: The control host is provided with a microphone and stereo headphones, and the main control board of the control host is integrated with a processor, a cellular network module, an image codec module and an audio codec module; The control host receives the audio data stream and the video data stream through the 4G / 5G network; The audio codec module decodes the received audio data stream and converts it into a playable audio signal, and the stereo headset plays the audio; The image encoding and decoding module decodes the received video data stream and converts it into displayable image frames, and the dual-diffraction optical machine component plays the image frames.

6. The multifunctional detection and recognition interactive system based on AR glasses according to claim 1 is characterized in that: The gesture recognition component is arranged on the right side of the bottom of the device host, and the shooting direction of the gesture recognition lens thereof faces downward to shoot the wearer's right hand gesture at a close distance.

7. A method for using the multifunctional detection and recognition interaction system based on AR glasses according to any one of claims 1 to 6, characterized in that: Here are the steps: Fix the headband to the head, make sure the device is in front of the forehead and the dual diffraction optical machine assembly is in front of the bridge of the nose, and the waveguide diffraction lens is facing the eyes; Start the control host and device host, and the system will self-check and initialize; The camera captures the real-life image of the surrounding environment, the TOF sensor measures the distance between the object and the sensor in real time, and the system analyzes the user's environment; The position of virtual content is automatically calculated based on the data, the corresponding content is retrieved, and the virtual image is superimposed on the user's real field of view through the waveguide diffraction lens; Adjust the focus of the thermal imaging lens via the manual focus ring; The thermal imaging lens assembly captures the thermal radiation of an object and converts it into a visual thermal image; The gesture recognition component captures and recognizes the user's gestures, and transmits the recognized gesture signals to the processing unit to output corresponding instructions or corresponding information; The control host communicates remotely with other control hosts or back-end platforms through the 4G / 5G network, including sending text messages and real-time audio and video broadcasts.

8. The method for using the multifunctional detection and recognition interactive system based on AR glasses according to claim 7, characterized in that: The gesture recognition component captures and recognizes user gestures, including: S1, read in the pre-trained palm detection model and hand key point detection model; S2, hand image captured by the gesture recognition camera; S3, using a palm detection model to locate the palm area in the hand image; S4, using the hand key point detection model to obtain the key point position of the hand; S5, using the position information of the key points, calculate the finger angle through an algorithm; S6. Determine the gesture type by comparing the calculated finger angle with a preset threshold.

9. The method for using the multifunctional detection and recognition interactive system based on AR glasses according to claim 8, characterized in that: The calculation of the finger angle in step 5 includes the following steps: S501, calculating a first vector from the wrist to the base of the finger; S502, calculating a second vector from the base of the finger to the fingertip; S503: Calculate the angle between the first vector and the second vector.

10. The method for using the multifunctional detection and recognition interactive system based on AR glasses according to claim 8, characterized in that: Step 6 includes the following steps: S601, setting the thumb angle threshold to 53°, the first angle threshold to 65°, and the second angle threshold to 49°; S602: If the angle of the thumb is greater than the thumb angle threshold, then proceed to step S603; otherwise, proceed to step S605; S603: If the angles of the remaining four fingers are greater than the first angle threshold, it is determined to be a fist gesture, otherwise, the process goes to step S604; S604, judging the remaining four fingers as numbers 1, 2, 3, and 4 according to the number of fingers whose angles are smaller than the second angle threshold; S605: If the angle of the thumb is smaller than the second angle threshold, and the angles of the other four fingers are larger than the first angle threshold, it is determined to be a thumb gesture, otherwise, the process goes to step S606; S606: If the angles of the thumb and the little finger are less than the second angle threshold, and the angles of the other three fingers are greater than the first angle threshold, it is determined to be the number 6, otherwise, the process goes to step S607; S607: If the angles of the thumb and index finger are less than the second angle threshold, and the angles of the other three fingers are greater than the first angle threshold, it is determined to be the number 8, otherwise, the process goes to step S608; S608: If the angles of the thumb, index finger, middle finger, ring finger, and pinky finger are all smaller than the second angle threshold, it is determined that the palm is open.

Images