Multifunctional detection and identification interaction system based on AR glasses and use method of multifunctional detection and identification interaction system

Abstract

The invention relates to a multifunctional detection and recognition interaction system based on AR glasses, which comprises a head band wearing part and a device host, a back plate of the device host is connected with the front part of the head band wearing part, the bottom of the device host is provided with a double-diffraction optical machine assembly and a gesture recognition assembly, and the double-diffraction optical machine assembly comprises an image output optical machine and a waveguide diffraction lens. The waveguide diffraction lens is used for imaging an image source to left and right eye view areas; the gesture recognition assembly is used for shooting and recognizing gestures of a wearer; a camera and a TOF sensor are arranged in the middle of the front side face of the device host. By using the image output light machine and the waveguide diffraction lens, a clearer image can be provided, the waveguide diffraction lens realizes a lighter and thinner design, virtual information can be naturally overlaid into the visual field of a user, and obvious visual distortion cannot be caused; a built-in gesture recognition assembly enables a user to send an instruction through a simple gesture; the camera and the TOF sensor are combined for use, and the functions of accurate space positioning, object distance measurement and the like are achieved.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a multifunctional detection and recognition interaction system based on AR glasses and a method for using the same. Background Art

[0002] In industrial environments, augmented reality (AR) glasses have received widespread attention as a tool that can provide real-time information and guidance. Some industrial work environments require wearing a helmet, such as power inspection in power plants, fire rescue at fire scenes, and surveying and mapping operations at construction sites; some industrial work environments do not require wearing a helmet, such as electronic maintenance in electronic assembly workshops and home appliance repairs.

[0003] Existing AR glasses products usually adopt a more traditional glasses structure design, in which the electronic components used to display virtual images, such as displays, optical components, processors and other electronic modules, are mainly concentrated in the middle of the two frames, and the weight distribution problem is not well solved, resulting in fatigue or discomfort after long-term wearing, especially the pressure on the bridge of the nose and ears. In addition, AR glasses with conventional glasses structures have less configurable space, resulting in them only providing basic display functions and lacking advanced interactive means such as gesture recognition and voice control, which limits their application potential in complex scenarios such as industry.

[0004] The above problems are worth solving. Summary of the invention

[0005] In order to overcome the deficiencies of the existing technology, the present invention provides a multifunctional detection and recognition interaction system based on AR glasses and a method of using the same.

[0006] The technical solution of the present invention is as follows: The present invention provides a multifunctional detection and recognition interaction system based on AR glasses, comprising a headband wearing piece and a device host, wherein the back plate of the device host is connected to the front lower edge of the headband wearing piece, and a double diffraction optical machine component and a gesture recognition component are arranged at the bottom of the device host, wherein the double diffraction optical machine component comprises an image output optical machine and a waveguide diffraction lens, wherein the image output optical machine is used to output an image source to the waveguide diffraction lens, and the waveguide diffraction lens is used to image the image source into the left and right eye fields of the wearer; the gesture recognition component is used to photograph and recognize the gestures of the wearer; a camera and a TOF sensor are arranged in the middle of the front side surface of the device host, wherein the camera is used to photograph a real-scene image of the surrounding environment, and the TOF sensor is used to measure the distance between an object and the TOF sensor in real time; The gesture recognition component captures and recognizes the wearer's gesture, including: S1, read in the pre-trained palm detection model and hand key point detection model; S2, hand image captured by the gesture recognition camera; S3, using a palm detection model to locate the palm area in the hand image; S4, using the hand key point detection model to obtain the key point position of the hand; S5, using the position information of the key points, calculate the finger angle through an algorithm; S501, calculating a first vector from the wrist to the base of the finger; S502, calculating a second vector from the base of the finger to the fingertip; S503, calculating the angle between the first vector and the second vector; S6. Determine the gesture type by comparing the calculated finger angle with a preset threshold.

[0007] According to the present invention of the above scheme, infrared LED fill lights are provided on both the left and right sides of the camera.

[0008] According to the above-mentioned scheme of the present invention, the working mainboard of the device host is integrated with a processing unit and a storage unit. The processing unit is used to analyze the environment according to the real scene and spatial positioning data obtained by the camera and / or sensor of the AR glasses, and automatically calculate the corresponding position for placing the virtual content, and is also used to retrieve the display content from the storage unit and display it at the corresponding position.

[0009] According to the present invention of the above scheme, a thermal imaging lens assembly is provided on the side of the device host, and the thermal imaging lens assembly is provided with a manual focus ring, and the focal length of the thermal imaging lens is adjusted by rotating the manual focus ring.

[0010] According to the above-mentioned scheme of the present invention, the working mainboard of the device host is integrated with a processing unit and a storage unit. The processing unit is used to analyze the environment according to the real scene and spatial positioning data obtained by the camera and / or sensor of the AR glasses, and automatically calculate the corresponding position for placing the virtual content, and is also used to retrieve the display content from the storage unit and display it at the corresponding position.

[0011] According to the above-mentioned solution of the present invention, the working mainboard of the device host is integrated with a positioning module and an inertial measurement module, the positioning is used to obtain the current position of the user, and the inertial measurement module is used to detect the movement state of the user.

[0012] The present invention according to the above scheme further includes a control host, which is electrically connected to the device host through a data line to transmit data, or sends a control instruction to the device host.

[0013] According to the present invention of the above scheme, the control host is provided with a pickup and stereo headphones, and the main control board of the control host is integrated with a processor, a cellular network module, an image codec module and an audio codec module; The control host receives the audio data stream and the video data stream through the 4G / 5G network; The audio codec module decodes the received audio data stream and converts it into a playable audio signal, and the stereo headset plays the audio; The image encoding and decoding module decodes the received video data stream and converts it into displayable image frames, and the dual-diffraction optical machine component plays the image frames.

[0014] According to the above-mentioned solution of the present invention, the gesture recognition component is arranged on the bottom right side of the device host, and the shooting direction of the gesture recognition lens thereof faces downward to shoot the wearer's right hand gesture at a close distance.

[0015] The present invention also provides a method for using the multifunctional detection and recognition interaction system based on AR glasses as described above: Fix the headband to the head, make sure the device is in front of the forehead and the dual diffraction optical machine assembly is in front of the bridge of the nose, and the waveguide diffraction lens is facing the eyes; Start the control host and device host, and the system will self-check and initialize, including the camera, infrared LED fill light, TOF sensor, gesture recognition component and thermal imaging lens; The camera captures the real-life image of the surrounding environment, the TOF sensor measures the distance between the object and the sensor in real time, and the system analyzes the user's environment; The position of virtual content is automatically calculated based on the data, the corresponding content is retrieved, and the virtual image is superimposed on the user's real field of view through the waveguide diffraction lens; Adjust the focus of the thermal imaging lens via the manual focus ring; The thermal imaging lens assembly captures the thermal radiation of an object and converts it into a visual thermal image; The gesture recognition component captures and recognizes the user's gestures, and transmits the recognized gesture signals to the processing unit to output corresponding instructions or corresponding information; The control host communicates remotely with other control hosts or back-end platforms through the 4G / 5G network, including sending text messages and real-time audio and video broadcasts.

[0016] According to the present invention of the above scheme, the gesture recognition component captures and recognizes the user gesture, including: S1, read in the pre-trained palm detection model and hand key point detection model; S2, hand image captured by the gesture recognition camera; S3, using a palm detection model to locate the palm area in the hand image; S4, using the hand key point detection model to obtain the key point position of the hand; S5, using the position information of the key points, calculate the finger angle through an algorithm; S6, judging the gesture type by comparing the calculated finger angle with a preset threshold; Furthermore, the calculation of the finger angle in step 5 includes the following steps: S501, calculating a first vector from the wrist to the base of the finger; S502, calculating a second vector from the base of the finger to the fingertip; S503: Calculate the angle between the first vector and the second vector.

[0017] Further, step 6 includes the following steps: S601, setting the thumb angle threshold to 53°, the first angle threshold to 65°, and the second angle threshold to 49°; S602: If the angle of the thumb is greater than the thumb angle threshold, then proceed to step S603; otherwise, proceed to step S605; S603: If the angles of the remaining four fingers are greater than the first angle threshold, it is determined to be a fist gesture, otherwise, the process goes to step S604; S604, judging the remaining four fingers as numbers 1, 2, 3, and 4 according to the number of fingers whose angles are smaller than the second angle threshold; S605: If the angle of the thumb is smaller than the second angle threshold, and the angles of the other four fingers are larger than the first angle threshold, it is determined to be a thumb gesture, otherwise, the process goes to step S606; S606: If the angles of the thumb and the little finger are less than the second angle threshold, and the angles of the other three fingers are greater than the first angle threshold, it is determined to be the number 6, otherwise, the process goes to step S607; S607: If the angles of the thumb and index finger are less than the second angle threshold, and the angles of the other three fingers are greater than the first angle threshold, it is determined to be the number 8, otherwise, the process goes to step S608; S608: If the angles of the thumb, index finger, middle finger, ring finger, and pinky finger are all smaller than the second angle threshold, it is determined that the palm is open.

[0018] Furthermore, in order to reduce the complexity of the model while ensuring the accuracy of the model, the palm detection model and the hand key point detection model of the present invention both adopt the DHRNet network model.

[0019] The present invention according to the above scheme has the following beneficial effects: The present invention can provide clearer images by using an image output optical machine and a waveguide diffraction lens. The waveguide diffraction lens of the dual diffraction optical machine assembly can achieve a lighter and thinner design and help to naturally superimpose virtual information into the user's field of view without causing obvious visual distortion or discomfort. The built-in gesture recognition component allows users to send commands and communicate through gestures through simple gestures; the combination of the equipped camera and TOF sensor not only allows the device to better understand the surrounding environment, but also realizes functions such as accurate spatial positioning and object distance measurement; The back panel of the device host is directly connected to the lower edge of the front end of the headband wearing piece, which helps to ensure comfort when worn for a long time and ensures a good user experience even in continuous working state. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a structural schematic diagram of the present invention; Figure 2 It is a structural schematic diagram of another perspective of the present invention; Figure 3 It is a structural exploded diagram of the present invention; Figure 4 It is a structural schematic diagram of the headband wearing piece in the present invention; Figure 5 This is a schematic diagram of the back structure of the device host in the present invention; Figure 6 It is a structural exploded diagram of the main unit of the device of the present invention; Figure 7 A schematic diagram of the tilted arrangement of the waveguide diffraction lens in the present invention; Figure 8 It is a structural exploded view of the double diffraction optical machine assembly in the present invention; Fig. 9 This is a structural breakdown diagram of the double-diffraction optomechanical assembly from another perspective; Fig.10 It is a structural schematic diagram of a thermal imaging lens assembly in a preferred embodiment; Fig.11 It is a structural diagram of the existing HRNet network model; Fig.12 It is a structural diagram of the DHRNet network model of the present invention.

[0021] In the figure, 1. headband wearing part; 101. front part; 102. support foot; 11. rotating shaft; 12. electrical socket; 13. concave arc structure; 14. first threading hole; 2. Device host; 20. Placement slot; 21. Accommodation chamber; 211. Chamber cover; 22. Axis slot; 221. Axis pin; 23. Arc slot; 24. Second threading hole; 3. Double diffraction optical machine assembly; 31. Image output optical machine; 32. Waveguide diffraction lens; 321. Arc-shaped notch; 33. Optical machine main housing; 331. Square heat dissipation hole; 332. Round light exit hole; 333. Mounting slot; 34. Housing side cover; 35. Housing back plate; 351. Limiting slot; 352. Convex arc surface; 353. Positioning pin; 41. Camera; 42. Infrared LED fill light; 43. TOF sensor; 44. Gesture recognition lens assembly; 5. Protective glasses; 51. Snap-in slot; 52. Supporting slot; 6. Thermal imaging lens assembly; 61. Manual focus ring; 62. Lens mount; 63. Plug connector; 7. Elastic band; 8. first power supply host; 81. connection hole; 82. concave arc surface; 9. Second power supply host; 91. Data cable. DETAILED DESCRIPTION

[0022] In order to better understand the purpose, technical scheme and technical effect of the present invention, the present invention is further explained in conjunction with the accompanying drawings and embodiments. It should be noted that similar numbers and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, it is stated that the embodiments described below are only used to explain the present invention and are not used to limit the present invention.

[0023] It should be noted that when an element is referred to as being "fixed on" or "disposed on" another element, it may be directly on the other element or there may also be a central element, and when an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time.

[0024] The indicated orientation or position relationship is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the product of the application is usually placed when used, or the orientation or position relationship commonly understood by technical personnel in this field, or the orientation or position relationship in which the product of the application is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0025] The terms "first" and "second" are only used for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly specifying the number of technical features.

[0026] like Figures 1 to 3As shown, a multifunctional detection and recognition interaction system based on AR glasses includes a headband wearing piece 1 and a device host 2. The headband wearing piece 1 and the device host 2 can be fixedly connected, rotatably connected, or detachably connected. The user fixes the device host 2 on the user's forehead by wearing the headband wearing piece 1.

[0027] In this embodiment, the back plate of the device host 2 is connected to the lower edge of the front portion 101 of the headband wearing part 1, so that the headband wearing part 1 can avoid the heat dissipation holes on the back of the device, and prevent the heat dissipation effect of the device host 2 from being affected by the obstruction. The device host 2 is connected to the lower edge of the headband wearing part 1, and the upper surface of the device host 2 can be located below the front portion 101 of the headband wearing part 1, so that the upper surface of the device host 2 forms a placement slot 20 in front of the headband wearing part 1. When the user wears a safety helmet, the brim of the safety helmet can be placed on the placement slot 20, the AR glasses can be used with the safety helmet, and the device host 2 can bear part of the weight of the safety helmet; when the user takes off the safety helmet, there is no need to take off the AR glasses. It can be seen that the use of AR glasses is independent of the safety helmet and can be firmly worn on the user's head.

[0028] The device host 2 is in the shape of a crescent, and the width of the device host 2 is equal to the distance between the two side legs 102 of the headband wearing piece 1, so that the size of the device host 2 is closer to the visor, increasing the contact area with the visor, which is conducive to the stability of wearing AR glasses and a helmet at the same time. The working mainboard inside the device host 2 is also in the shape of a crescent to adapt to the internal structure of the device host 2, and at the same time has sufficient space for the layout of electronic components, providing a physical basis for the realization of multifunctional AR glasses.

[0029] The top of the front portion 101 of the headband wearing piece 1 is tilted backwards, so that the front portion 101 of the headband wearing piece 1 can be smoothly inserted into the gap between the safety helmet and the wearer's head, and the back side of the front portion 101 of the headband wearing piece 1 can fit the forehead of the user (or wearer), thereby increasing the wearing contact area, reducing the gravity pressure of the device and the safety helmet, and improving wearing comfort.

[0030] The front part 101 and the two side legs 102 of the headband wearing piece 1 are integrally formed to form a ring with an opening, and the two side legs 102 have a certain elasticity, which is convenient for the user to open the two legs 102 and wear. An elastic band 7 is provided at the opening of the headband wearing piece 1, and the two ends of the elastic band 7 are respectively connected to the two side legs 102, and the elastic band 7 is used to adjust the tightness of the headband wearing piece 1.

[0031] like Figure 5As shown, in a preferred embodiment, a rotating shaft 11 is provided at the lower edge of the front portion 101 of the headband wearing part 1, an axis groove 22 is provided on the back of the device main body 2, an axis pin 221 is provided in the axis groove 22, and the rotating shaft 11 is rotatably connected with the axis pin, so that the device main body 2 can rotate along the rotating shaft 11 relative to the headband wearing part 1, and the rotation angle is 0 to 60°.

[0032] like Figure 4 As shown, in an optional embodiment, the connection between the front portion 101 of the headband wearing part 1 and the support leg 102 has a concave arc structure 13, which can reduce the weight of the front portion 101 of the headband wearing part 1, which is beneficial to reduce the front end weight of the AR glasses product as a whole, and avoid the front end of the product being too heavy and affecting the wearing stability and comfort. The back side of the headband wearing part 1 (the side in contact with the user's head) is provided with a flexible material such as silicone or sponge to prevent the hard material shell from causing damage to the user's head, so the flexible back side of the headband wearing part 1 can improve wearing comfort and safety.

[0033] In the present invention, an electrical socket 12 is provided at the end of one of the legs 102 of the headband wearable component 1, and the electrical socket 12 is electrically connected to the working mainboard of the device host 2; the first power supply host 8 is plugged into the electrical socket 12 using a power cord to provide power to the working mainboard. Among them, a flexible circuit board is provided inside the headband wearable component 1, and a wiring channel is provided at the connection between the headband wearable component 1 and the device host 2. One end of the flexible circuit board is connected to the electrical socket 12, and the other end is electrically connected to the working mainboard through a connecting wire in the wiring channel.

[0034] The output interface of the first power supply host 8 is arranged on a side close to the electrical socket 12, so as to facilitate connecting the output interface of the first power supply host 8 and the electrical socket 12 of the headband wearing component 1 on the same side, thereby shortening the length of the connecting wire.

[0035] In a specific embodiment, the front shell of the headband wearing part 1 is provided with a first threading hole 14, and the shell back plate 35 of the device host 2 is correspondingly provided with a second threading hole 24, the first threading hole 14 serves as one end of a wiring channel between the headband wearing part 1 and the device host 2, and the second threading hole 24 serves as the other end of the wiring channel, and the wiring channel has a built-in connection line for connecting the flexible circuit board and the working main board. In an AR glasses product in which the headband wearing part 1 and the device host 2 are a rotating connection structure, the first threading hole 14 is provided on the side of the front shell of the headband wearing part 1 close to the rotating shaft 11, and the second threading hole 24 is provided on the side wall of the shaft groove 22, and the first threading hole 14 and the second threading hole 24 are located on the same side of the rotating shaft 11.

[0036] The first power supply mainframe 8 is provided with through holes 81 at both ends, and the elastic band 7 is detachably connected to the through holes 81 to install the first power supply mainframe 8 on the elastic band 7. Specifically, one end of the elastic band 7 is connected to the end of the left leg 102, and the other end passes through the left through hole 81 of the first power supply mainframe 8 and the right through hole 81 of the first power supply mainframe 8 in sequence, and then connected to the end of the right leg 102. The bottom surface of the first power supply mainframe 8 (the side close to the user's head) is a concave arc surface 82, and the concave arc surface structure design is conducive to better fitting the back of the user's head and improving wearing comfort.

[0037] In the present invention, a dual-diffraction optical machine component 3 is provided in the middle of the bottom of the device host 2. The dual-diffraction optical machine component 3 includes an image output optical machine 31 and a waveguide diffraction lens 32. The image output optical machine 31 is used to output the image source to the waveguide diffraction lens 32. The waveguide diffraction lens 32 is used to image the image source into the left and right eye fields of view of the wearer, thereby achieving a left and right dual-channel augmented reality effect.

[0038] In one application example, the image source of the image output optical machine 31 comes from the cloud or the backend terminal. Specifically, in the firefighting industry, the command center monitors the on-site environment through the camera 41, analyzes the on-site fire situation and formulates a rescue plan at the backend terminal, and then remotely sends data such as commands or analysis information to the AR glasses of the on-site firefighters, and finally images the image to the waveguide diffraction lens 32 through the dual-diffraction optical machine component 3, so that the firefighters can obtain information synchronously and communicate with the command center in real time.

[0039] In other application examples, the image source of the image output optical machine 31 comes from the storage unit in the device host 2. The working mainboard of the device host 2 integrates at least a processing unit (MCU) and a storage unit. The processing unit can analyze the environment according to the real scene and spatial positioning data obtained by the camera 41 and / or sensor of the AR glasses, and automatically calculate how to accurately place the virtual content in the corresponding position of the real world, retrieve the display content from the storage unit and display it at the corresponding position. For example, if the position of the cabinet in front and the distance from the user are analyzed to be 2 meters, the text information "2 meters" and the distance ruler icon are correctly displayed on the cabinet.

[0040] like Figure 8 and Fig. 9As shown, in an optional embodiment, the dual-diffraction optical machine assembly 3 includes not only the image output optical machine 31 and the waveguide diffraction lens 32, but also includes: an optical machine main shell 33, a shell side cover 34, and a shell back plate 35. The optical machine main shell 33 has a chamber for installing the image output optical machine 31, and the chamber has a lateral opening. The shell side cover 34 can be detachably installed at the lateral opening of the optical machine main shell 33. The optical machine main shell 33 and the shell side cover 34 are assembled into a protective shell. A square heat dissipation hole 331 is provided on one side of the optical machine main shell 33, and a circular light exit hole 332 is provided on the other side opposite thereto, and the square heat dissipation hole 331 and the circular light exit hole 332 are located on both sides of the lateral opening. The square heat dissipation hole 331 can provide a heat dissipation outlet for the image output optical machine 31 in the protective shell, which is beneficial to the discharge of heat generated by the image output optical machine 31 during operation. The size of the circular light outlet 332 is equal to the size of the light port of the image output optical machine 31, which can provide a larger light outlet for the image output optical machine 31 in the protective shell, and can meet the requirement of the image output optical machine 31 to use a shorter focal length lens to achieve the same field of view, which is beneficial for projecting the image onto the waveguide diffraction lens 32 at a close distance; not only that, the larger light outlet can pass more light, which helps to improve the overall brightness of the picture, thereby providing a clear virtual image.

[0041] The protective shell assembled by the main shell 33 of the optical machine and the side cover of the shell is provided with a mounting groove 333 for mounting the waveguide diffraction lens 32 on the side close to the circular light exit hole 332, and the shell back plate 35 is correspondingly provided with a limiting groove 351, and the limiting groove 351 is adapted to the structure of the waveguide diffraction lens 32. The shell back plate 35 can be installed and removed from the protective shell, and the waveguide diffraction lens 32 is fixed by the mutually spliced ​​limiting groove 351 and the mounting groove 333. Screw holes are provided at the upper and lower ends of the shell back plate 35. When maintaining or replacing the waveguide diffraction lens 32, the shell back plate 35 can be removed by screwing the screws, and then the waveguide diffraction lens 32 can be removed. When installing the waveguide diffraction lens 32, the waveguide diffraction lens 32 is fixed by the combined structure of the limiting groove 351 and the mounting groove 333, without the need for complicated lens debugging, which is very convenient.

[0042] There are positioning pins 353 on the sides of the upper and lower screw holes of the shell back plate 35, and there are corresponding pin holes on the back of the optical machine main shell 33. When installing the shell back plate 35, you only need to align the positioning pins 353 and insert them into the pin holes to quickly complete the installation.

[0043] An arc-shaped notch 321 is provided at the lower middle side of the waveguide diffraction lens 32. The width of the middle lens only needs to cover the circular light exit hole 332 so that the light can be smoothly coupled into the waveguide diffraction lens 32. The arc-shaped notch 321 can not only play the role of installation positioning, but also reduce the lens material to save costs. Correspondingly, a convex arc surface 352 is provided at the lower end of the limiting groove 351 of the shell back plate 35. The convex arc surface 352 is in contact with the lower surface of the arc-shaped notch 321 of the waveguide diffraction lens 32 to ensure that the waveguide diffraction lens 32 is installed stably.

[0044] like Figure 7 As shown, in a preferred embodiment, the waveguide diffraction lens 32 is tilted, specifically, its top end is close to the image output optical machine 31, and its bottom end is far away from the image output optical machine 31; the tilt angle α of the waveguide diffraction lens 32 ranges from 10° to 15°, and the tilted waveguide diffraction lens 32 can not only make the light of the image output optical machine 31 better coupled into the waveguide diffraction lens 32, but also prevent the ambient light under the AR glasses from being reflected to the human eye through the inner mirror surface of the waveguide diffraction lens 32, affecting the viewing of the image on the waveguide diffraction lens 32. Accordingly, the mounting slot 333 of the optical machine main housing 33 and the limiting slot 351 of the housing back plate 35 are both tilted to adapt to the waveguide diffraction lens 32 in the tilted state.

[0045] like Figure 5 and Figure 6 As shown, in an optional embodiment, the middle of the bottom of the housing of the device main body 2 extends downward to form a housing chamber 21 for installing the double diffraction optical machine component 3, and the inner cavity of the device main body 2 is connected to the housing chamber 21, so that the wiring row of the double diffraction optical machine component 3 can be passed from the housing chamber 21 into the inner cavity of the device main body 2 and then be electrically connected to the working main board. The housing chamber 21 is equipped with a detachable chamber cover 211, and two studs are arranged inside the housing chamber 21. The chamber cover 211 is correspondingly provided with bolt holes. The main housing 33 of the double diffraction optical machine component 3 is provided with a connecting ear with a screw hole. The double diffraction optical machine component 3 is placed in the housing chamber 21, and the chamber cover 211 is covered. The screw holes of the studs, the screw holes of the connecting ears, and the bolt holes of the chamber cover 211 are aligned accordingly. The bolts pass from the outside to the inside. The double diffraction optical machine component 3 can be fixedly installed in the housing chamber 21 by tightening the bolts. The side walls of the accommodating cavity 21 and the side walls of the cavity cover plate 211 are both provided with notches for avoiding the waveguide diffraction lens 32 .

[0046] In the present invention, a transparent protective mirror 5 is provided on the bottom front side of the device main body 2. The material of the protective mirror 5 can be polycarbonate. The polycarbonate protective mirror 5 has the characteristics of impact resistance and light weight. The material of the protective mirror 5 can also be glass. The glass protective mirror 5 has a higher hardness, a harder and smoother surface, and is not easily scratched.

[0047] The bottom of the protective glasses 5 is provided with a supporting groove 52. When the user wears the AR glasses, the bridge of the nose of the user is placed in the supporting groove 52, which increases the contact area between the bridge of the nose and the protective glasses 5, reduces the gravitational pressure of the protective glasses 5 on the bridge of the nose, and improves wearing comfort. Preferably, a flexible material such as silicone or sponge can be provided at the bottom of the supporting groove 52.

[0048] The protective goggles 5 are arc-shaped, with a chord height of not less than 2 cm, and a width greater than the width of the waveguide diffraction lens 32, to provide sufficient space for installing the double-diffraction optical-mechanical assembly 3 to avoid affecting wearing; at the same time, the protective goggles 5 form a semi-enclosed protective structure on the front side of the double-diffraction optical-mechanical assembly 3.

[0049] like Figure 5 As shown, an arc-shaped slot 23 is provided at the bottom of the shell of the device host 2 corresponding to the position of the protective mirror 5, a snap-in block is provided on the inner wall of the shell adjacent to the arc-shaped slot 23, and a snap-in slot 51 is provided at the top of the protective mirror 5. The number and position of the snap-in slots 51 correspond one-to-one to those of the snap-in blocks. Through the above structure, the protective mirror 5 and the device host 2 can be detachably installed, which is convenient for maintenance personnel to repair or replace the aged protective mirror 5.

[0050] like Figure 1 As shown, in the present invention, a camera 41 is provided in the middle of the front side of the device host 2. The camera 41 is used to capture the real-life image of the surrounding environment, and the processing unit performs calculation processing to realize the spatial positioning and three-dimensional mapping of the surrounding environment. Based on the real-world image captured by the camera 41, the AR glasses can overlay virtual information or graphics on it in real time. The shooting data obtained by the camera 41 can be stored in the storage unit, and can also be transmitted to other devices or uploaded to the cloud through a wireless network or a mobile network.

[0051] Infrared LED fill lights 42 are provided on both sides of the camera 41. In a completely dark environment, the infrared light emitted by the infrared LED fill lights 42 illuminates the object, and the CCD or CMOS sensor in the camera 41 captures the reflected infrared light and converts it into an image, thereby realizing the infrared night vision function of the camera 41.

[0052] A TOF sensor 43 is also provided in the middle of the front side of the device host 2, and the TOF sensor 43 is located below the camera 41; the TOF sensor 43 is used to measure the distance between the object and the TOF sensor 43 in real time; the TOF sensor 43 can measure a distance value for each point in the field of view, thereby generating a complete depth map. The depth map shows the depth information of each position, that is, the actual distance corresponding to each pixel. The generated depth map will be further processed, and the AR glasses system will reconstruct the three-dimensional environment around the user through the depth map, and place virtual objects or perform other operations accordingly.

[0053] A thermal imaging lens assembly 6 is provided on the side of the device host 2. The thermal imaging lens assembly 6 generates images by detecting infrared radiation (heat) emitted by objects, and can display the temperature distribution of different objects. Through the temperature information captured by the thermal imaging lens, the AR glasses system can overlay virtual information in the user's field of view, such as temperature readings, alarm prompts, etc.

[0054] In a specific application example, AR glasses with thermal imaging function play a very important role in the field of firefighting. Thermal imaging lenses can quickly detect and display high-temperature areas, helping firefighters quickly locate the source of the fire. In some cases, the source of the fire may be blocked by walls, furniture or other obstacles. Thermal imaging lenses can detect heat through obstacles to help firefighters find hidden sources of fire, which is conducive to firefighters taking measures earlier; through thermal imaging images, firefighters can intuitively see the direction and speed of the spread of the fire, so as to better plan fire-fighting strategies and evacuation routes; in smoke and low-visibility environments, using the obvious difference between the heat emitted by the human body and the surrounding environment, thermal imaging lenses help firefighters find trapped people, making rescue work more efficient; after the fire is extinguished, thermal imaging lenses can be used to detect locations where embers or hot spots may exist to prevent re-ignition, etc.

[0055] like Fig.10 As shown, in a preferred embodiment, the thermal imaging lens assembly 6 is provided with a manual focus ring 61, and the focus of the thermal imaging lens can be adjusted by rotating the manual focus ring 61 to obtain the desired clarity. The clarity of the thermal image directly affects the user's ability to identify and analyze details. Taking detail recognition as an example, a clear image can help the user more accurately identify and distinguish different objects or areas. For example, in industrial inspection, a clear image can more easily find hot spots or defects in the equipment. In addition, because the autofocus system may be affected in certain extreme environments (such as high temperature, high humidity, low light, etc.), the thermal imaging lens with manual focus function is more reliable in high-temperature fire scenes.

[0056] In an optional embodiment, the thermal imaging lens assembly 6 has a lens seat 62, a plug connector 63 is provided on one side of the lens seat 62, and the device host 2 is correspondingly provided with a plug slot, and the thermal imaging lens assembly 6 is detachably connected to the device host 2 through the connection structure between the plug connector 63 and the plug slot, and power is obtained through the device host 2. This embodiment satisfies the user to select AR glasses products according to actual needs, and has higher flexibility; it is also conducive to reducing maintenance costs. When the thermal imaging lens fails or is damaged, the thermal imaging lens assembly 6 can be simply replaced instead of the entire device.

[0057] In the present invention, the working mainboard of the device host 2 is integrated with a Beidou module and / or a GPS module and an inertial measurement module. The Beidou module and / or the GPS module obtains the user's current precise location information by receiving Beidou or GPS satellite signals; the inertial measurement module is used to detect the user's motion state, including walking direction, speed and posture.

[0058] The AR glasses system accesses a map database in the cloud or a storage unit, calculates the optimal path based on the user's current location and destination, and generates navigation instructions; the AR glasses system displays navigation instructions, such as navigation arrows, distance prompts, etc., through the waveguide diffraction lens 32, and finally realizes superimposing navigation information in the user's field of view.

[0059] In the present invention, the AR glasses are equipped with a second power supply host 9, and the optional size of the second power supply host 9 is 115×70×34 mm, which is convenient for the operator to place the second power supply host 9 in an arm pocket or a chest pocket. The second power supply host 9 is connected to the electrical socket 12 of the device host 2 via a data cable 91.

[0060] In an optional embodiment, the first power supply host 8 is a control host, and the second power supply host 9 is a mobile charging power source.

[0061] In another optional embodiment, the first power supply host 8 is a mobile charging power supply, and the second power supply host 9 is a control host.

[0062] Among them, the mobile charging power supply is connected to the device host 2 through the electrical socket 12 of the headband wearable component 1 to provide the working mainboard with the required power for operation. The control host is connected to the device host 2 through the electrical socket 12 of the headband wearable component 1, which can not only supply power to the device host, but also send and receive data with the device host 2 and send control instructions to the device host 2. Specifically, the control host is provided with a microphone and stereo headphones. The main control board of the control host integrates a processor, a cellular network module (4G / 5G), an image codec module and an audio codec module. The processor is used to coordinate the work of each module to process video and audio data. The control host receives the audio and video data of the other party through the 4G / 5G network; the audio codec module decodes the received audio data stream and converts it into a playable audio signal, and the stereo headphones play the audio; the image codec module decodes the received video data stream and converts it into a displayable image frame, and the image output optical machine 31 outputs the image frame.

[0063] The camera 41 of the device host 2 captures the video image in front of the user, and transmits the video data stream to the control host after encoding through the processing unit; the control host collects the user's voice through the microphone and encodes it to form an audio data stream; the control host sends the audio and video data to the other party through the 4G / 5G network. It can be seen that the present invention increases the voice and video call function of AR glasses by configuring the control host, and sends the image in front of the call to the other party through the network connection, similar to the live broadcast method, and can communicate in real time.

[0064] In an optional embodiment, the control host is provided with a display screen or a touch screen, and a plurality of buttons are provided below the screen. The display content includes, but is not limited to: network operator, network connection status, battery power, date, time and text messages. The buttons include four direction keys of up, down, left and right and a confirmation key, which facilitates setting operations, selection operations, page turning operations, etc. on the control host. When the screen is a touch screen, the user can perform the above operations through the touch screen.

[0065] like Figure 2 As shown, in the present invention, a gesture recognition lens assembly 44 is provided at the bottom of the device host 2 close to the right side of the human hand, and the shooting direction of the gesture recognition lens assembly 44 is downward. It can be seen that the AR glasses gesture recognition area of ​​the present solution is on the right-hand side below the AR glasses, and the right-hand gesture is photographed at a close distance. This design is in line with the right-hand habit of most people.

[0066] The present invention also provides a method for using the multifunctional detection and recognition interaction system based on AR glasses of the above solution: Step A, fix the headband to the head, ensure that the device host is located in front of the forehead and the dual diffraction optical machine component is located in front of the bridge of the nose, and the waveguide diffraction lens is facing the eyes; Step B: Start the control host and the device host, and the system self-checks and initializes, including the camera, infrared LED fill light, TOF sensor, gesture recognition component and thermal imaging lens; Step C: The camera captures the real-life image of the surrounding environment, the TOF sensor measures the distance between the object and the sensor in real time, and the system analyzes the user's environment; Step D: automatically calculating the position of the virtual content according to the data, calling up the corresponding content, and superimposing the virtual image into the user's real field of view through the waveguide diffraction lens; Step E: Adjust the focus of the thermal imaging lens by manually adjusting the focus ring; Step F, the thermal imaging lens assembly captures the thermal radiation of the object and converts it into a visual thermal image; Step G: The gesture recognition component captures and recognizes the user's gesture, and transmits the recognized gesture signal to the processing unit to output corresponding instructions or corresponding information; Step H: The control host remotely communicates with other control hosts or back-end platforms through the 4G / 5G network, including sending text messages and real-time audio and video broadcasts.

[0067] In step G, the gesture recognition component captures and recognizes the user's gesture, including: S1, read in the pre-trained palm detection model and hand key point detection model; S2, hand image captured by the gesture recognition camera; S3, using a palm detection model to locate the palm area in the hand image; S4, using the hand key point detection model to obtain the key point position of the hand; S5, using the position information of the key points, calculate the finger angle through an algorithm; S6, judging the gesture type by comparing the calculated finger angle with a preset threshold; The calculation of the finger angle in step S5 includes the following steps: S501, calculating a first vector from the wrist to the base of the finger; S502, calculating a second vector from the base of the finger to the fingertip; S503: Calculate the angle between the first vector and the second vector.

[0068] Wherein, step S6 comprises the following steps: S601, setting the thumb angle threshold to 53°, the first angle threshold to 65°, and the second angle threshold to 49°; S602: If the angle of the thumb is greater than the thumb angle threshold, then proceed to step S603; otherwise, proceed to step S605; S603: If the angles of the remaining four fingers are greater than the first angle threshold, it is determined to be a fist gesture, otherwise, the process goes to step S604; S604, judging the remaining four fingers as numbers 1, 2, 3, and 4 according to the number of fingers whose angles are smaller than the second angle threshold; S605: If the angle of the thumb is smaller than the second angle threshold, and the angles of the other four fingers are larger than the first angle threshold, it is determined to be a thumb gesture, otherwise, the process goes to step S606; S606: If the angles of the thumb and the little finger are less than the second angle threshold, and the angles of the other three fingers are greater than the first angle threshold, it is determined to be the number 6, otherwise, the process goes to step S607; S607: If the angles of the thumb and index finger are less than the second angle threshold, and the angles of the other three fingers are greater than the first angle threshold, it is determined to be the number 8, otherwise, the process goes to step S608; S608: If the angles of the thumb, index finger, middle finger, ring finger, and pinky finger are all smaller than the second angle threshold, it is determined that the palm is open.

[0069] In order to reduce the complexity of the model while ensuring the accuracy of the model, the palm detection model and the hand key point detection model of the present invention both adopt the DHRNet network model, which is improved on the existing HRNet network model.

[0070] like Fig.11 As shown in the figure, first, the HRNet network model downsamples the input image through two convolution operations, reducing the resolution of the input image by 4 times to 1 / 4 of the initial size, and then enters the main network. The main part of the HRNet network consists of four stages, among which stage 1 is composed of 4 BottleNeck residual units. After stage 1, there is a series of exchange structures and stage structures. Each time a network passes through an exchange structure, a scale branch is added through a downsampling operation. In addition, in order to obtain multi-scale information, each stage in the network will fuse features of different scales. Stages 2, 3, and 4 are stacked repeatedly 1, 4, and 3 times respectively. The last exchange unit in stage 4 only retains the feature layer with the highest resolution, and finally predicts the hand points on the high-resolution feature map.

[0071] By observing the HRNet network structure, we can find that with the increase of low-resolution sub-networks, the number of network branches and convolution modules also increases, which leads to an exponential increase in the number of network parameters and the computational overhead of the model as the order increases. However, in practical applications, both high speed and high accuracy are often required.

[0072] like Fig.12 As shown in the figure, the DHRNet network model retains the first three stages (stage 1, stage 2 and stage 3), and changes stage 4 to stage D. Stage D fuses multi-scale features through transposed convolution and greatly reduces the number of model parameters. It also adds another transposed convolution to the highest resolution feature map of its output to generate a higher resolution feature map for predicting heat maps.

[0073] In order to better handle scale changes and improve the network's ability to perceive learning errors, a multi-resolution supervision strategy is used during training. Specifically, the highest resolution (1 / 4 resolution of the input) branch in stage D and the higher resolution (1 / 2 resolution of the input) branch after transposed convolution are used to calculate the loss function of the predicted heat map. The final heat map loss value is the sum of the losses of the above two branches.

[0074] In stage D, the 1 / 4 resolution heat map containing i key points and the final 1 / 2 resolution heat map after the transposed convolution operation are the learning targets of the network. The real heat map h is generated using a two-dimensional Gaussian distribution function. i (x,y):

[0075] Among them, (x, y) represents the position coordinates of any pixel in the real heat map, (x i ,y i ) is the center of the Gaussian function, represents the true position coordinates of the ith key point, and represents the standard deviation of the Gaussian function preset for each key point.

[0076] Prediction Heatmap It is calculated by the following formula:

[0077] The loss function is divided into classification loss and regression loss, and the mean square error MSE loss function is used for calculation. The formula is as follows:

[0078] Among them, Nb represents the number of hand key points, and They represent the predicted heat map and the true heat map of the b-th heat map respectively.

[0079] The highest resolution (1 / 4 resolution of the input) branch loss value in stage D is used Indicates that the loss value of the higher resolution branch (1 / 2 resolution of the input) after transposed convolution is used Indicates that the total loss value L of the final key point detection heat map is:

[0080] Finally, experiments are provided: the improved DHRNet network is experimentally verified on the COCO dataset, as follows: 1. The COCO key point detection dataset designs the target key point similarity (OKS) indicator to indicate the similarity between the true value and the predicted hand key points. The OKS calculation method is as follows:

[0081] Among them, i represents the i-th hand key point, d i represents the Euclidean distance between the predicted position and the true position of the ith hand key point, s represents the scale of the target, is the area of ​​the target detection box, k i Represents the normalization factor of the hand point, which is a constant used to control the attenuation. i Indicates the visibility of the hand point, which can be 0, 1, or 2. 0 means the hand point is not labeled, 1 means the hand point is labeled but is not visible due to occlusion, and 2 means the hand point is labeled and visible. Indicates when The value is 1 when , otherwise it is 0.

[0082] 2. Average accuracy AP:

[0083] Among them, p represents the target number, T is the preset OKS threshold, and only when the OKS value is greater than the threshold, it indicates that the prediction is correct. Used to represent the relationship between OKS and threshold T. If OKS is greater than T, the value is 1.

[0084] It represents the accuracy when T=n. represents the average accuracy of medium-scale targets, represents the average accuracy of large-scale targets, represents the average recall rate.

[0085] First, the aspect ratio of the hand detection box height to width is adjusted to 4:3, and then the box is cropped from the image and adjusted to a fixed input size. In order to reduce computational cost and experimental cycle, the input size of this experimental network is set to 256 pixels x 192 pixels. In addition, the data augmentation method adopts random rotation, random scale transformation and flipping, with the rotation range of [-45°, 45°] and the random scale transformation range of [0.65, 1.35]. It should be noted that 0.65 and 1.35 have no units, they represent the scale factor relative to the original image size.

[0086]

[0087] Experimental results show that improving stage 4 to stage D can effectively reduce the number of network parameters and calculations, and the accuracy is basically the same as the original one.

[0088] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described.

[0089] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multifunctional detection and recognition interactive system based on AR glasses, characterized in that: The device comprises a headband wearing piece and a main device, wherein the back plate of the main device is connected to the front lower edge of the headband wearing piece, and the bottom of the main device is provided with a double diffraction optical machine component and a gesture recognition component, wherein the double diffraction optical machine component comprises an image output optical machine and a waveguide diffraction lens, wherein the image output optical machine is used to output an image source to the waveguide diffraction lens, and the waveguide diffraction lens is used to image the image source into the left and right eye fields of view of the wearer; and the gesture recognition component is used to capture and recognize the gestures of the wearer; A camera and a TOF sensor are provided in the middle of the front side of the device host, the camera is used to capture the real-life image of the surrounding environment, and the TOF sensor is used to measure the distance between the object and the TOF sensor in real time; The gesture recognition component captures and recognizes the wearer's gesture, including: S1, read in the pre-trained palm detection model and hand key point detection model; S2, hand image captured by the gesture recognition camera; S3, using a palm detection model to locate the palm area in the hand image; S4, using the hand key point detection model to obtain the key point position of the hand; S5, using the position information of the key points, calculate the finger angle through an algorithm; S501, calculating a first vector from the wrist to the base of the finger; S502, calculating a second vector from the base of the finger to the fingertip; S503, calculating the angle between the first vector and the second vector; S6. Determine the gesture type by comparing the calculated finger angle with a preset threshold.

2. The multifunctional detection and recognition interactive system based on AR glasses according to claim 1 is characterized in that: A thermal imaging lens assembly is provided on the side of the device host, and the thermal imaging lens assembly is provided with a manual focus ring, and the focal length of the thermal imaging lens is adjusted by rotating the manual focus ring.

3. The multifunctional detection and recognition interactive system based on AR glasses according to claim 1 is characterized in that: The working mainboard of the device host is integrated with a positioning module and an inertial measurement module. The positioning is used to obtain the current position of the user, and the inertial measurement module is used to detect the movement state of the user.

4. The multifunctional detection and recognition interactive system based on AR glasses according to claim 1 is characterized in that: It also includes a control host, which is electrically connected to the device host through a data line to transmit data, or sends a control instruction to the device host.

5. The multifunctional detection and recognition interactive system based on AR glasses according to claim 4 is characterized in that: The control host is provided with a microphone and stereo headphones, and the main control board of the control host is integrated with a processor, a cellular network module, an image codec module and an audio codec module; The control host receives the audio data stream and the video data stream through the 4G / 5G network; The audio codec module decodes the received audio data stream and converts it into a playable audio signal, and the stereo headset plays the audio; The image encoding and decoding module decodes the received video data stream and converts it into displayable image frames, and the dual-diffraction optical machine component plays the image frames.

6. The multifunctional detection and recognition interactive system based on AR glasses according to claim 1 is characterized in that: The gesture recognition component is arranged on the right side of the bottom of the device host, and the shooting direction of the gesture recognition lens thereof faces downward to shoot the wearer's right hand gesture at a close distance.

7. A method for using the multifunctional detection and recognition interaction system based on AR glasses according to any one of claims 1 to 6, characterized in that: Here are the steps: Fix the headband to the head, make sure the device is in front of the forehead and the dual diffraction optical machine assembly is in front of the bridge of the nose, and the waveguide diffraction lens is facing the eyes; Start the control host and device host, and the system will self-check and initialize; The camera captures the real-life image of the surrounding environment, the TOF sensor measures the distance between the object and the sensor in real time, and the system analyzes the user's environment; The position of virtual content is automatically calculated based on the data, the corresponding content is retrieved, and the virtual image is superimposed on the user's real field of view through the waveguide diffraction lens; Adjust the focus of the thermal imaging lens via the manual focus ring; The thermal imaging lens assembly captures the thermal radiation of an object and converts it into a visual thermal image; The gesture recognition component captures and recognizes the user's gestures, and transmits the recognized gesture signals to the processing unit to output corresponding instructions or corresponding information; The control host communicates remotely with other control hosts or back-end platforms through the 4G / 5G network, including sending text messages and real-time audio and video broadcasts.

8. The method for using the multifunctional detection and recognition interactive system based on AR glasses according to claim 7, characterized in that: The gesture recognition component captures and recognizes user gestures, including: S1, read in the pre-trained palm detection model and hand key point detection model; S2, hand image captured by the gesture recognition camera; S3, using a palm detection model to locate the palm area in the hand image; S4, using the hand key point detection model to obtain the key point position of the hand; S5, using the position information of the key points, calculate the finger angle through an algorithm; S6. Determine the gesture type by comparing the calculated finger angle with a preset threshold.

9. The method for using the multifunctional detection and recognition interactive system based on AR glasses according to claim 8, characterized in that: The calculation of the finger angle in step 5 includes the following steps: S501, calculating a first vector from the wrist to the base of the finger; S502, calculating a second vector from the base of the finger to the fingertip; S503: Calculate the angle between the first vector and the second vector.

10. The method for using the multifunctional detection and recognition interactive system based on AR glasses according to claim 8, characterized in that: Step 6 includes the following steps: S601, setting the thumb angle threshold to 53°, the first angle threshold to 65°, and the second angle threshold to 49°; S602: If the angle of the thumb is greater than the thumb angle threshold, then proceed to step S603; otherwise, proceed to step S605; S603: If the angles of the remaining four fingers are greater than the first angle threshold, it is determined to be a fist gesture, otherwise, the process goes to step S604; S604, judging the remaining four fingers as numbers 1, 2, 3, and 4 according to the number of fingers whose angles are smaller than the second angle threshold; S605: If the angle of the thumb is smaller than the second angle threshold, and the angles of the other four fingers are larger than the first angle threshold, it is determined to be a thumb gesture, otherwise, the process goes to step S606; S606: If the angles of the thumb and the little finger are less than the second angle threshold, and the angles of the other three fingers are greater than the first angle threshold, it is determined to be the number 6, otherwise, the process goes to step S607; S607: If the angles of the thumb and index finger are less than the second angle threshold, and the angles of the other three fingers are greater than the first angle threshold, it is determined to be the number 8, otherwise, the process goes to step S608; S608: If the angles of the thumb, index finger, middle finger, ring finger, and pinky finger are all smaller than the second angle threshold, it is determined that the palm is open.

Images