Intelligent walking backward glasses

Abstract

The application discloses intelligent reverse walking glasses, comprising: a frame, which is closed, and has a lens; a panoramic camera, which is installed at the rear end of the frame, and is used for collecting images of the surrounding environment of a wearer; a dynamic vision sensor, which is used for capturing moving objects in a weak light environment; an inertial measurement unit, which is installed at the middle part of the frame, and is used for measuring the angular velocity and acceleration of the intelligent reverse walking glasses; a miniature VR projection module, which is installed at the front part of the frame close to the lens, and is used for projecting an obstacle avoidance direction signal to the lens; a vibration sensor, which is installed at the tail of the frame, and is used for providing a warning signal for the wearer; and a processor, which receives images collected by the panoramic camera, addresses and information of changed pixels collected by the dynamic vision sensor, and angular velocity collected by the inertial measurement unit, controls the vibration sensor to provide a warning signal, calculates an obstacle avoidance direction signal, and transmits the obstacle avoidance direction signal to the miniature VR projection module.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of photoelectric sensing and display technology, in particular to a smart reverse walking glasses. BACKGROUND

[0002] Reverse walking as a new fitness method, in recent years, is deeply loved by the public. Reverse walking has great benefits in medical function, which can enhance the balance ability of the exerciser, relieve waist pain, stretch the knee ligament and prevent knee joint degeneration. Studies have shown that long-term adherence to reverse walking exercise has significant effect on relieving waist pain and preventing adolescent humpback.

[0003] In the existing technical solutions, most of them use narrow field of view cameras to directly observe the situation behind the exerciser, and measure the distance between the rear obstacle and the exerciser through infrared sensors, and the buzzer gives an early warning to avoid obstacles.

[0004] In the process of implementing the present application, the inventors found that at least the following problems exist in the prior art:

[0005] On the one hand, the exerciser needs to carry multiple sensors, which increases the exercise load; on the other hand, the narrow field of view camera has a limited shooting range, and the early warning mechanism only has the function of reminding to stop, and cannot realize the functions of avoiding obstacles without turning the head and path planning. At the same time, the dynamic objects (running pedestrians, moving bicycles, etc.) that quickly approach the exerciser in complex environments have not been effectively detected. In addition, in the scene with dark ambient light, the image quality obtained by the camera is poor, and the scheme of relying on camera projection display observation is not reliable. In addition, the field of view of the traditional camera is limited and cannot fully observe the environment around the exerciser. SUMMARY

[0006] The purpose of the embodiments of the present application is to provide a smart reverse walking glasses to solve the technical problems of complex device design, small camera field of view coverage, and difficulty in application in dark light and complex environments in the related art, and to realize the technical innovation of guiding the user to avoid the rear static obstacles and early warning of fast dynamic approaching objects.

[0007] According to a first aspect of the embodiments of the present application, a smart reverse walking glasses is provided, comprising:

[0008] a frame, the frame is closed, and the frame has a lens;

[0009] a panoramic camera, the panoramic camera is installed at the rear end of the frame and is used to collect images of the surrounding environment of the wearer;

[0010] a dynamic vision sensor, the dynamic vision sensor is used to capture moving objects in a weak light environment;

[0011] An inertial measurement unit is installed in the middle of the frame to measure the angular velocity and acceleration of the smart anti-falling glasses;

[0012] A miniature VR projection module is installed in the front of the frame near the lens to project the obstacle avoidance direction signal to the lens;

[0013] A vibration sensor is installed at the tail of the frame to provide a warning signal for the wearer;

[0014] A processor receives the images collected by the panoramic camera, the address and information of the changed pixels collected by the dynamic vision sensor, and the angular velocity collected by the inertial measurement unit, controls the vibration sensor to provide a warning signal, calculates the obstacle avoidance direction signal and transmits it to the miniature VR projection module.

[0015] Further, a mechanical adjustment knob is provided on the frame for adjusting the size of the frame.

[0016] Further, a GNSS is also included, which is installed in the middle of the frame to locate the wearer and record motion data.

[0017] Further, a miniature microphone array is installed in the front of the frame, and a miniature speaker is installed at the tail of the frame. The miniature microphone array transmits the wearer's voice instructions to the processor, and the processor issues voice replies through the miniature speaker.

[0018] Further, the processing process of the processor includes:

[0019] Obtain the panoramic event stream data collected by the dynamic vision sensor under the panoramic lens, integrate it into a panoramic event frame image within a certain time window, and put it into an event processing queue;

[0020] Obtain the gray-scale image output by the gray-scale channel of the panoramic event camera, expand the omnidirectional gray-scale image into a panoramic image by equidistant cylindrical projection, and put it into a depth prediction queue;

[0021] Send the panoramic image to the trained depth prediction neural network and the trained ground flatness prediction neural network to obtain the depth map and the position points of the uneven ground;

[0022] Obtain the data of the inertial measurement unit, use the frequency of the inertial measurement unit as the minimum event window, calculate the pose change of adjacent event frames, filter out the event points generated by the motioner's own motion, and obtain the event points of moving objects;

[0023] The filtered event frame is sent into the trained event target detection neural network to obtain a panoramic event frame sequence with dynamic targets marked;

[0024] The depth map, the position points of the ground unevenness and the panoramic event frame sequence with dynamic targets marked are real-time aligned to calculate the depth of the fast-moving target, and when the depth value is less than a predetermined threshold, the vibration sensor is triggered to vibrate to prewarn the wearer;

[0025] According to the depth map, a three-dimensional depth overhead curve is drawn, wherein the maximum value of the curve is the farthest position from the observation point, the uneven part of the ground in the curve is shielded, and a safe retreat range depth overhead curve is obtained;

[0026] An obstacle avoidance direction is calculated, and the micro VR projection module is used to display the direction on the lens.

[0027] Further, the processing process of the processor further includes:

[0028] After the movement ends, the movement data recorded by the inertial measurement unit and the GNSS are visualized on the lens through the VR projection module, and in addition, the step frequency, step length and step count information of the mover can be further calculated and displayed on the lens through the micro VR projection module.

[0029] Further, the processing process of the processor further includes:

[0030] User instructions can be obtained from the micro microphone array, and voice replies can be generated by the built-in AI voice control robot and output from the micro speaker.

[0031] The technical scheme provided by the embodiment of the application can include the following beneficial effects:

[0032] As known from the above embodiment, the panoramic camera is used to capture images, a single panoramic camera can perform large field of view perception on the environment around the wearer, which is more conducive to ensuring the safety of the mover; the dynamic vision sensor is used, which can still work normally in a dark environment, has the advantages of low delay, high dynamic range and high time resolution, can accurately capture fast-moving objects, and only produces a response when there is a moving object in the picture, and has low power consumption; the inertial measurement unit is used to remove the motion of the glasses themselves and the interference of the event points generated on the dynamic vision sensor; the micro VR projection module is used to output the display information to the lens of the glasses, which is intuitive and visual for the user and friendly to use.

[0033] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0035] Figure 1 The figure is a schematic structural diagram of smart backward walking glasses according to an exemplary embodiment.

[0036] Figure 2 The figure is a flowchart showing the function implementation of smart reverse walking glasses according to an exemplary embodiment.

[0037] Reference numerals:

[0038] 1. Frame; 2. Panoramic camera; 3. Dynamic vision sensor; 4. Inertial measurement unit; 5. Micro VR projection module; 6. Vibration sensor; 7. Processor; 8. Mechanical adjustment knob; 9. GNSS; 10. Micro microphone array; 11. Micro speaker; 12. Lens. DETAILED DESCRIPTION

[0039] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0040] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0041] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0042] Figure 1 FIG. 1 is a schematic structural diagram of a pair of smart backward-walking glasses according to an exemplary embodiment. Figure 1As shown, the intelligent backward walking glasses can include a frame 1, a panoramic camera 2, a dynamic vision sensor 3, an inertial measurement unit 4, a miniature VR projection module 5, a vibration sensor 6 and a processor 7. The frame 1 is closed, and the frame 1 has a lens 12. The panoramic camera 2 is installed at the rear end of the frame 1, and is used to collect images of the surrounding environment of the wearer. The dynamic vision sensor 3 is used to capture moving objects in a weak light environment. The inertial measurement unit 4 is installed at the middle of the frame 1, and is used to measure the angular velocity and acceleration of the intelligent backward walking glasses. The miniature VR projection module 5 is installed at the front of the frame 1 close to the lens 12, and is used to project the obstacle avoidance direction signal to the lens 12. The vibration sensor 6 is installed at the tail of the frame 1, and is used to provide a warning signal for the wearer. The processor 7 receives the images collected by the panoramic camera 2, the address and information of the changed pixels collected by the dynamic vision sensor 3, and the angular velocity and acceleration collected by the inertial measurement unit 4, controls the vibration sensor 6 to provide a warning signal, calculates the obstacle avoidance direction signal and transmits it to the miniature VR projection module 5.

[0043] As can be seen from the above embodiment, the panoramic camera 2 is used to capture images, and a single panoramic camera 2 can perceive the environment around the wearer with a large field of view, which is more conducive to ensuring the safety of the wearer. The dynamic vision sensor 3 is used, which can still work normally in a dark environment, has the advantages of low delay, high dynamic range and high time resolution, can accurately capture fast-moving objects, and only responds when there are moving objects in the picture, and has low power consumption. The inertial measurement unit 4 is used, which can remove the motion of the glasses itself and the interference of the event points generated on the dynamic vision sensor 3. The miniature VR projection module 5 is used, which outputs the display information to the glasses lens 12, and the user can intuitively see and use it friendly.

[0044] Algorithm: Implement depth estimation, event target detection and ground flatness prediction algorithm (the innovation in the algorithm is not very big, mainly the integration of existing advanced algorithms)

[0045] User interaction: voice interaction, vibration reminder, VR projection lens 12 display (display obstacle avoidance direction and user motion data that can be obtained by the following supplementary hardware)

[0046] Hardware supplement: To filter out event information generated by the motion of the device itself and further record motion data, IMU and RNSS sensors are added

[0047] Specifically, the panoramic camera 2 adopts a panoramic ring lens with a 360°x(30°-100°) panoramic angle, a vertical half field of view of 100°, and an effective vertical half field of view of 70°, which can capture the environment behind for depth estimation and capture the ground conditions to detect the ground flatness and guide the user to avoid pits and bumps. In specific implementations, the panoramic camera 2 can be selected by those skilled in the art according to the needs, which is a conventional technical means in the art and will not be described here.

[0048] In specific implementations, the vibration sensor 6 can be installed at the tail of the frame 1 and attached to the user's ear bone near the temple position, so that slight vibrations can be easily felt.

[0049] Specifically, the frame 1 can also be provided with a mechanical adjustment knob 8 for adjusting the size of the frame 1. In an embodiment, the mechanical gear is clamped on a stretchy band that is in contact with the gear groove (the stretchy band material is generally soft plastic or other new materials), so that the back part is stretched to better fit the human brain, and the design is based on ergonomics.

[0050] Specifically, the processing process of the processor 7 can include:

[0051] Step S11: Obtain the panoramic event stream data collected by the dynamic vision sensor 3 under the panoramic lens, integrate it into a panoramic event frame image within a certain time window, and put it into an event processing queue;

[0052] Specifically, the processing operation of integrating event points into event frames is a method commonly used in the art and will not be described here. Using the form of event frames, event information can be represented in a form similar to images, which is convenient for subsequent algorithm processing; the integrated event frame can filter out noise event points to some extent.

[0053] Step S12: Obtain the gray-scale image output by the gray-scale channel of the panoramic event camera, expand the omnidirectional gray-scale image into a panoramic image by equidistant cylindrical projection, and put it into a depth prediction queue;

[0054] Specifically, equidistant cylindrical projection of omnidirectional images is also a common operation in data preprocessing for panoramic depth prediction, and will not be described here. The processing into the display form of our common rectangular pictures is convenient for subsequent algorithm processing.

[0055] In specific implementations, steps S11 and S12 can be executed in parallel to improve the overall reading and calculation rate of the device.

[0056] Step S13: Send the panoramic image into the trained depth prediction neural network and the trained ground flatness prediction neural network to obtain the depth map and the position points of the ground unevenness;

[0057] Specifically, the depth prediction neural network can be ADENet, RectNet, Bi-Fuse, etc. In the embodiment, a U-Net architecture is adopted, and unsupervised training is performed through a binocular disparity loss function, i.e., a view reconstruction loss, a disparity consistency loss, and a disparity smoothness loss. An open-source OmniDepth and 3D60 panoramic depth estimation dataset is used as the dataset.

[0058] The ground flatness prediction neural network can be yolov5. In the embodiment, a yolov5 architecture is adopted, pre-training parameters are loaded for training, and fine-tuning is performed in an open-source RTK road surface condition dataset. The network backend outputs three classification heads, i.e., “flat”, “relatively flat”, and “pitted”. The class with the highest confidence is selected as the ground prediction result, and the value thereof is used as the ground quality evaluation score. If the output class is “pitted”, the ground pitted position is marked with an anchor box in the figure. In specific implementation, the pre-training parameters are usually model parameters pre-trained on ImageNet. Such a design has high recognition accuracy and high speed: yolov5 is the most accurate and fastest technical solution based on an RGB image at present.

[0059] 1. Flip the panoramic image up and down (the reason is that the panoramic camera is inverted, and the image captured is upside down)

[0060] Preferably, the panoramic image input into the neural network can be cropped to remove the invalid field of view blocked by the wearer's head, and the attention area is limited to the part outside the human eye field of view, so as to reduce the data amount, improve the operation speed, and meet the requirement of real-time performance.

[0061] Step S14: Obtain the data of the inertial measurement unit 4, take the frequency of the inertial measurement unit 4 as the minimum event window, calculate the pose change of adjacent event frames, filter out event points generated due to the motion of the mover, and obtain event points of a moving object;

[0062] Specifically, the PnP method is used to calculate the pose change of adjacent event frames.

[0063] Step S15: Input the filtered event frame into the trained event target detection neural network to obtain a panoramic event frame sequence in which dynamic targets are marked;

[0064] Specifically, the event target detection neural network is a network architecture of an event processing front end (event scatter point integration into a framed image) + yolov5, and the pre-training parameters are loaded for training, and fine-tuning is performed in a small sample until convergence.

[0065] Preferably, the event frame sent into the neural network can be cropped to remove the invalid field of view blocked by the wearer's head, and the attention area can be limited to the part outside the human eye field of view, so as to reduce the data amount, improve the operation speed, and meet the requirement of real-time performance.

[0066] Step S16: real-time alignment of the depth map, the position points of the ground unevenness, and the panoramic event frame sequence marking the dynamic target, and calculation of the depth of the fast-moving target, triggering the vibration sensor 6 to vibrate when the depth value is less than a predetermined threshold, to prewarn the wearer.

[0067] Specifically, the predetermined threshold is an empirical value, and the depth prediction of the classical vision algorithm is relatively reliable within 10 m, and is set to 3 m in this embodiment.

[0068] Step S17: drawing a three-dimensional depth overhead curve according to the depth map, wherein the maximum value of the curve is the position farthest from the observation point, and the ground unevenness is shielded in the curve to obtain a safe retreat range depth overhead curve.

[0069] Specifically, the shielding method is to directly crop the part with obstacles in the horizontal coordinate to obtain an updated depth overhead curve.

[0070] Preferably, the 180° field of view not covered by the human eye field of view can be equally divided into 6 parts, one part for each 30°, and the safe movement direction within the 30° range is represented by only one direction arrow, and the direction arrow is displayed on the lens 12 to guide the user to avoid obstacles.

[0071] Step S18: calculation of the obstacle avoidance direction, and display on the lens 12 by means of the micro-VR projection module 5.

[0072] Specifically, the obstacle avoidance direction is the direction of the line connecting the maximum value of the safe retreat range depth overhead curve obtained in step S17 and the position of the observation point (coordinate origin of the safe retreat depth range overhead curve).

[0073] Specifically, the intelligent backward walking glasses can further include a GNSS 9 installed in the middle part of the frame 1 for positioning the wearer and recording motion data. In a specific implementation, the inertial measurement unit 4 and the GNSS 9 are installed in the middle part of the frame 1, which can appropriately avoid the disturbance of the sensor data caused by the wearer wearing the glasses sliding; by using the inertial measurement unit 4 and the GNSS 9, the rotation matrix R and the translation motion vector t of the device motion can be solved, and then the pose of the device at the frame time can be solved, effectively filtering out the event information generated by the camera's own motion, and providing accurate trajectory positioning of the mover.

[0074] Correspondingly, the processing process of the processor 7 can further include:

[0075] Step S21: after the movement, the movement data recorded by the inertial measurement unit 4 and the GNSS 9 are visualized on the lens 12 through the VR projection module;

[0076] Step S22: the step frequency, stride length and step count information of the runner are calculated and displayed on the lens 12 through the micro VR projection module 5.

[0077] The specific implementation of steps S21-S22 is a routine practice in the art, which is not described here.

[0078] In particular, the intelligent backward walking glasses can further include a micro microphone array 10 installed at the front of the frame 1 and a micro speaker 11 installed at the tail of the frame 1, the micro microphone array 10 transmits the voice instructions of the wearer to the processor 7, and the processor 7 issues voice replies through the micro speaker 11.

[0079] In a specific implementation, the micro microphone array 10 is arranged at the front of the frame 1 close to the mouth of the wearer to reduce noise interference in sound transmission, and the micro speaker 11 is arranged close to the ear of the wearer to ensure the high definition of the output sound. Voice interaction is adopted to interactively complete the switching of the display information of the lens 12, and the customized display requirements of the user are met, and the simple interaction mode effectively ensures the safety of the runner.

[0080] Correspondingly, the processing process of the processor 7 can further include:

[0081] The user instructions can be obtained from the micro microphone array 10, and voice replies are generated by the built-in AI voice control robot and output from the micro speaker 11.

[0082] The specific implementation of this step is also a routine design in the art, which is not described here.

[0083] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains.

[0084] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application.

Claims

1. A pair of smart backward walking glasses, characterized in that: The application relates to a smart reverse walking glasses, which comprises the following parts: a frame, which is closed and has lenses; a panoramic camera, which is installed at the rear end of the frame and used for collecting images of the surroundings of a wearer; a dynamic vision sensor, which is used for capturing moving objects in a weak light environment; an inertial measurement unit, which is installed at the middle part of the frame and used for measuring the angular velocity and acceleration of the smart reverse walking glasses; a miniature VR projection module, which is installed at the front part of the frame close to the lenses and used for projecting an obstacle avoidance direction signal onto the lenses; a vibration sensor, which is installed at the tail of the frame and used for providing a warning signal for the wearer; a processor, which receives images collected by the panoramic camera, addresses and information of changed pixels collected by the dynamic vision sensor and angular velocity collected by the inertial measurement unit, controls the vibration sensor to provide a warning signal, calculates an obstacle avoidance direction signal and transmits the signal to the miniature VR projection module. The processing procedure of the processor comprises the following steps: acquiring panoramic event stream data collected by the dynamic vision sensor under the panoramic camera, integrating the data into panoramic event frame images within a certain time window and putting the images into an event processing queue; acquiring a gray image output by a gray channel of the panoramic event camera, expanding the omnidirectional gray image into a panoramic image through equidistant cylindrical projection and putting the image into a depth prediction queue; sending the panoramic image into a trained depth prediction neural network and a trained ground flatness prediction neural network to obtain a depth map and position points of ground unevenness; acquiring data of the inertial measurement unit, taking the frequency of the inertial measurement unit as a minimum event window, calculating the pose change of adjacent event frames, filtering out event points caused by the motion of a person himself and obtaining event points of moving objects; sending the filtered event frames into a trained event target detection neural network to obtain a panoramic event frame sequence in which dynamic targets are marked; aligning the depth map, the position points of ground unevenness and the panoramic event frame sequence in which dynamic targets are marked in real time to calculate the depth of a fast-moving target, and triggering the vibration sensor to vibrate when the depth value is less than a predetermined threshold to warn the wearer.

2. The smart walk-back glasses of claim 1, wherein, The processing procedure of the processor further comprises the following steps: drawing a three-dimensional depth overhead view curve according to the depth map, wherein the maximum value of the curve represents the position farthest from an observation point, the uneven part of the ground in the curve is shielded and a safe retreating range depth overhead curve is obtained; calculating an obstacle avoidance direction and displaying the direction on the lenses by means of the miniature VR projection module, wherein the obstacle avoidance direction is the direction of the line connecting the maximum value of the safe retreating range depth overhead curve and the observation point.

3. The intelligent walk backwards glasses according to claim 1, wherein, The frame is provided with a mechanical adjusting knob for adjusting the size of the frame.

4. The intelligent walk backwards glasses of claim 1, wherein, The application further comprises a GNSS, which is installed at the middle part of the frame and used for positioning the wearer and recording motion data.

5. The intelligent walk backwards glasses of claim 1, wherein, Also included are a miniature microphone array mounted on the front of the frame and a miniature speaker mounted on the tail of the frame, the miniature microphone array transmitting the wearer's voice instructions to the processor, the processor issuing voice replies through the miniature speaker.

6. The intelligent walk backwards glasses according to claim 4, wherein, The processor's processing process further includes: After the end of the movement, the motion data recorded by the inertial measurement unit and the GNSS are visualized on the lens through the VR projection module, and the step frequency, step length and step number information of the mover are calculated and displayed on the lens with the help of the miniature VR projection module.

7. The intelligent walk backwards glasses according to claim 5, wherein, The processor's processing process further includes: User instructions are obtained from the miniature microphone array, voice replies are generated by the built-in AI voice control robot, and the voice replies are output from the miniature speaker.

Images