Intelligent glasses system based on self-adaptive vision correction and multi-modal interaction and implementation method thereof

Abstract

The invention discloses an intelligent glasses system based on self-adaptive vision correction and multi-mode interaction and an implementation method thereof.The intelligent glasses system comprises a glasses frame, a loudspeaker, a solar power supply module, liquid crystal variable-focus lenses, an energy storage module, a data acquisition module and a data processing module, and the method comprises the steps that 1, data acquisition is conducted; step 2, vision detection and correction; step 3, performing multi-mode health monitoring; step 4, performing man-machine interaction; 5, emotion recognition and management; step 6, intelligent fatigue prevention; step 7, augmented reality; step 8, energy management; according to the invention, wavefront sensing and liquid crystal variable-focus lenses are adopted, so that vision monitoring and automatic correction are realized; comprehensive health management is provided by monitoring multiple physiological parameters; natural interaction is realized based on gesture and voice recognition; fusing the expression and the voice to recognize an emotional state and providing suggestions; detecting fatigue based on eye movement features, and adjusting parameters; navigation, translation and object recognition services are provided; a solar power supply module is additionally arranged to prolong the endurance time.

Description

Technical Field

[0001] The present invention relates to the technical field of wearable devices, and in particular to a smart glasses system based on adaptive vision correction and multimodal interaction and an implementation method thereof. Background Art

[0002] Smart glasses are a wearable device that combines advanced technology with glasses. It integrates technologies from multiple fields such as optics, electronics, and computers, and aims to provide users with convenient and efficient information interaction and functional experience. Smart glasses can realize a variety of functions, such as displaying information, taking photos and videos, navigating and positioning, etc., which brings great convenience to people's lives and work. However, the existing smart glasses still have the following shortcomings in actual use: First, they cannot meet the needs of users with impaired vision, resulting in users needing to wear additional corrective glasses, so they are not convenient to use; second, they can only monitor limited physiological parameters, so they cannot provide comprehensive health management; third, they need to be operated by touching buttons or using specific voices, so they are not natural and smooth enough in terms of human-computer interaction; fourth, they lack the perception of the user's emotional state and cannot provide personalized emotional management services; fifth, the display parameters cannot be intelligently adjusted according to the user's eye movement characteristics, so it is easy to cause eye fatigue; sixth, they can only provide simple navigation functions, which cannot meet the user's needs for a rich experience; seventh, they use traditional battery power supply methods, resulting in a short battery life and the need for frequent charging, which brings inconvenience to users. Summary of the invention

[0003] The purpose of the present invention is to provide a smart glasses system based on adaptive vision correction and multimodal interaction and an implementation method thereof, so as to solve the problems raised in the above background technology.

[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a smart glasses system based on adaptive vision correction and multimodal interaction, comprising a frame, a speaker, a solar power supply module, a liquid crystal variable focus lens, an energy storage module, a data acquisition module and a data processing module, wherein the data processing module is installed on the frame, the data processing module is electrically connected to the speaker, the solar power supply module, the liquid crystal variable focus lens and the data acquisition module, and the speaker, the solar power supply module, the liquid crystal variable focus lens and the data acquisition module are all installed on the frame.

[0005] Preferably, the data acquisition module includes a wavefront sensor, a biosensor, a front camera, a rear camera, a microphone, an eye movement sensor and a light sensor, and the biosensor includes an electrocardiogram sensor, a photoelectric volume pulse wave sensor, a skin electrode sensor and a temperature sensor.

[0006] Preferably, the data processing module includes a vision detection and correction unit, a multimodal health monitoring unit, a human-computer interaction unit, an emotion recognition and management unit, an intelligent anti-fatigue unit, an augmented reality unit and an energy management module.

[0007] Preferably, the human-computer interaction unit includes a gesture recognition subunit, a voice content recognition subunit and a command fusion subunit, and the emotion recognition and management unit includes a facial expression recognition subunit, a voice emotion recognition subunit, an emotion state determination subunit and an emotion management suggestion subunit.

[0008] Preferably, the augmented reality unit includes a real-time navigation subunit, a real-time translation subunit and an object recognition subunit.

[0009] The method for implementing a smart glasses system based on adaptive vision correction and multimodal interaction includes step one, data collection; step two, vision detection and correction; step three, multimodal health monitoring; step four, human-computer interaction; step five, emotion recognition and management; step six, intelligent anti-fatigue; step seven, augmented reality; step eight, energy management;

[0010] In the above step 1, the data collection module is used to collect human body data and environmental data;

[0011] In the above step 2, the data acquisition module transmits the wavefront data collected by the wavefront sensor to the vision detection and correction unit, and the vision detection and correction unit automatically adjusts the refractive index of the liquid crystal variable focus lens;

[0012] In the above step 3, the data acquisition module transmits the electrocardiogram signal, pulse wave signal, skin electrical signal and body temperature collected by the biosensor to the multimodal health monitoring unit, and the multimodal health monitoring unit automatically outputs the health status assessment result;

[0013] In the above step 4, the data acquisition module transmits the hand image collected by the rear camera and the voice signal collected by the microphone to the human-computer interaction unit, the gesture recognition subunit analyzes the hand image and recognizes the gesture command, the voice content recognition subunit analyzes the voice signal and recognizes the voice command, and the command fusion subunit fuses the recognized gesture and voice command to control the function of the system;

[0014] In the above step 5, the data acquisition module transmits the facial image collected by the front camera and the voice signal collected by the microphone to the emotion recognition and management unit, the facial expression recognition subunit analyzes the facial image and recognizes the facial expression, the voice emotion recognition subunit analyzes the voice signal and recognizes the voice emotion, the emotion state determination subunit performs weighted fusion recognition of the facial expression and the voice emotion and outputs the emotion recognition result, and the emotion management suggestion subunit generates the emotion management suggestion according to the emotion recognition result;

[0015] In the above step 6, the data acquisition module transmits the eyeball position and pupil diameter data collected by the eye movement sensor to the intelligent anti-fatigue unit, and the intelligent anti-fatigue unit determines the degree of fatigue and adjusts the display brightness and blue light intensity according to the degree of fatigue;

[0016] In the above step 7, the data acquisition module transmits the image data collected by the rear camera to the augmented reality unit, the real-time navigation subunit analyzes the image data, updates the three-dimensional point cloud map, the real-time translation subunit analyzes the image data, identifies the text area, and outputs the translation result, and the object recognition subunit analyzes the image data, identifies the object, and outputs the object category and position boundary box;

[0017] In the above step eight, the data acquisition module transmits the light intensity data collected by the light sensor to the energy management module, and the energy management module predicts the output P of the solar power supply module. solar (t), and monitor the power consumption P of each module in real time i (t), and use optimization algorithms to dynamically adjust the energy supply of each module to solve the following problems:

[0018]

[0019] The constraints are:

[0020] P total (t)≤P solar (t)+P battery (t).

[0021] Among them, P total (t) is the total energy consumption, P battery (t) is the electrical energy stored in the energy storage module.

[0022] Preferably, in step 2, the vision detection and correction unit uses Zernike polynomials to perform fitting calculation on the wavefront aberration, and the formula is as follows:

[0023]

[0024] Among them, ρ and θ are polar coordinates, is the Zernike polynomial, is the corresponding coefficient;

[0025] The formula for focal length is as follows:

[0026]

[0027] Wherein, f is the focal length, n(V) is the refractive index corresponding to the driving voltage V, and R is the radius of curvature of the lens.

[0028] Preferably, in step three, the multimodal health monitoring unit extracts heart rate, blood oxygen saturation and skin conductivity change rate from the electrocardiogram signal, pulse wave signal, skin electrical signal and body temperature, and then uses a multi-parameter fusion algorithm to fuse the heart rate, blood oxygen saturation, skin conductivity change rate and body temperature to obtain a feature vector, and finally inputs the feature vector into a support vector machine for health status classification;

[0029] The heart rate HR calculation formula is:

[0030]

[0031] in, is the mean RR interval;

[0032] Blood oxygen saturation SpO2 is calculated using the ratio R of red light and infrared light of the pulse wave signal:

[0033] SpO2=AB×R

[0034] Where A and B are calibration coefficients;

[0035] The calculation formula of skin conductivity change rate ΔGSR is as follows:

[0036]

[0037] Among them, GSR(t) is the skin electrical signal.

[0038] Preferably, in step six, the intelligent anti-fatigue unit extracts the blinking frequency and the eye movement speed, and determines the fatigue degree through a threshold value;

[0039] Blink frequency f blink The calculation formula is:

[0040]

[0041] Eye movement speed v t The calculation formula is:

[0042]

[0043] The display brightness L is calculated as follows:

[0044] L=L0×(1-αF)

[0045] The calculation formula of blue light intensity B is:

[0046] B=B0×(1-βF)

[0047] Among them, α and β are adjustment coefficients.

[0048] Preferably, in step eight, the charging and discharging strategy adopted by the energy management module is:

[0049] When P solar (t)>P total At (t), the solar power supply module charges the energy storage module;

[0050] When P solar (t)<P total (t), the energy storage module supplies power.

[0051] Compared with the prior art, the beneficial effects of the present invention are: the present invention adopts wavefront sensing and liquid crystal variable focus lenses to realize vision monitoring and automatic correction; provides comprehensive health management by monitoring multiple physiological parameters; realizes natural interaction based on gesture and voice recognition; integrates facial expressions and voice recognition emotional states and provides suggestions; detects fatigue and adjusts parameters based on eye movement characteristics; provides navigation, translation and object recognition services; and adds a solar power supply module to extend the battery life. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the three-dimensional structure of the frame of the present invention;

[0053] Figure 2 It is the overall structural diagram of the present invention;

[0054] Figure 3 It is a structural block diagram of the data acquisition module of the present invention;

[0055] Figure 4 is a structural block diagram of the biosensor of the present invention;

[0056] Figure 5 It is a structural block diagram of the data processing module of the present invention;

[0057] Figure 6 It is a structural block diagram of the human-computer interaction unit of the present invention;

[0058] Figure 7 This is a structural block diagram of the emotion recognition and management unit of the present invention;

[0059] Figure 8 This is a structural block diagram of the augmented reality unit of the present invention;

[0060] Fig. 9 is a flow chart of the vision detection and correction method of the present invention;

[0061] Fig.10 is a flow chart of the multimodal health monitoring method of the present invention;

[0062] Fig.11 It is a flow chart of the human-computer interaction method of the present invention;

[0063] Fig.12 This is a flow chart of the emotion recognition and management method of the present invention;

[0064] Fig.13 This is a flow chart of the intelligent anti-fatigue method of the present invention.

[0065] Fig.14 The figure is a flow chart of the implementation method of the present invention.

[0066] In the figure: 1. Frame; 11. Speaker; 12. Solar power supply module; 13. Liquid crystal variable focus lens; 14. Energy storage module; 2. Data acquisition module; 21. Wavefront sensor; 22. Biosensor; 221. ECG sensor; 222. Photoelectric volume pulse wave sensor; 223. Skin electricity sensor; 224. Temperature sensor; 23. Front camera; 24. Rear camera; 25. Microphone; 26. Eye movement sensor; 27. Light sensor; 3. Data processing module; 31. Vision detection and correction unit; 32. Multimodal health Health monitoring unit; 33. Human-computer interaction unit; 331. Gesture recognition subunit; 332. Voice content recognition subunit; 333. Command fusion subunit; 34. Emotion recognition and management unit; 341. Facial expression recognition subunit; 342. Voice emotion recognition subunit; 343. Emotion state determination subunit; 344. Emotion management suggestion subunit; 35. Intelligent anti-fatigue unit; 36. Augmented reality unit; 361. Real-time navigation subunit; 362. Real-time translation subunit; 363. Object recognition subunit; 37. Energy management module. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Please refer to the attached Figure 1 -Attached Figure 8The present invention provides an embodiment: a smart glasses system based on adaptive vision correction and multimodal interaction, comprising a frame 1, a speaker 11, a solar power supply module 12, a liquid crystal variable focus lens 13, an energy storage module 14, a data acquisition module 2 and a data processing module 3, wherein the frame 1 is provided with the data processing module 3, the data processing module 3 is electrically connected with the speaker 11, the solar power supply module 12, the liquid crystal variable focus lens 13 and the data acquisition module 2, and the speaker 11, the solar power supply module 12, the liquid crystal variable focus lens 13 and the data acquisition module 2 are all installed on the frame 1, the frame 1 is the main frame part of the smart glasses, and the speaker 11 is used to play speech. The solar power supply module 12 is used to power other modules and charge the energy storage module 14. The liquid crystal variable focus lens 13 is used to carry image information and can automatically zoom. The energy storage module 14 is used to store electrical energy and power the system. The data acquisition module 2 is used to collect human body data and environmental data. The data processing module 3 is used to analyze and process data; the data acquisition module 2 includes a wavefront sensor 21, a biosensor 22, a front camera 23, a rear camera 24, a microphone 25, an eye movement sensor 26 and a light sensor 27. The biosensor 22 includes an electrocardiogram sensor 221, a photoelectric volume pulse wave sensor 222, a skin electricity sensor 223 and a temperature sensor. The sensor 224 includes a wavefront sensor 21 for collecting wavefront data, an electrocardiogram sensor 221 for collecting electrocardiogram signals, a photoelectric volumetric pulse wave sensor 222 for collecting pulse wave signals, an electrocardiogram signal skin electrical sensor 223 for collecting skin electrical signals, a temperature sensor 224 for collecting temperature, a front camera 23 for collecting facial image information, a rear camera 24 for collecting environmental image information, a microphone 25 for collecting user voice signals, an eye movement sensor 26 for collecting eyeball position and pupil diameter data, and a light sensor 27 for collecting light intensity data; the data processing module 3 includes a vision detection and correction unit 31, a multimodal health monitoring unit 32, and a plurality of other components. The control unit 32, the human-computer interaction unit 33, the emotion recognition and management unit 34, the intelligent anti-fatigue unit 35, the augmented reality unit 36 ​​and the energy management module 37, the vision detection and correction unit 31 is used to automatically adjust the focal length of the liquid crystal variable focus lens 13, the multimodal health monitoring unit 32 is used to monitor the health of the human body, the human-computer interaction unit 33 is used to provide human-computer interaction services, the emotion recognition and management unit 34 is used to identify the user's emotions and make suggestions, the intelligent anti-fatigue unit 35 is used to intelligently adjust the display parameters of the liquid crystal variable focus lens 13, the augmented reality unit 36 ​​is used to provide a virtual and real interactive function, and the energy management module 37 is used for intelligent management of electric energy;The human-computer interaction unit 33 includes a gesture recognition subunit 331, a voice content recognition subunit 332 and a command fusion subunit 333. The emotion recognition and management unit 34 includes a facial expression recognition subunit 341, a voice emotion recognition subunit 342, an emotion state determination subunit 343 and an emotion management suggestion subunit 344. The gesture recognition subunit 331 is used to recognize gesture commands, the voice content recognition subunit 332 is used to recognize voice commands, the command fusion subunit 333 is used to accommodate gesture commands and voice commands, so as to control the system function, and the facial expression recognition subunit 341 is used to recognize facial expressions, the voice emotion recognition subunit 342 is used to recognize voice emotions, the emotional state determination subunit 343 is used to judge the user's emotions by integrating facial expressions and voice emotions, and the emotional management suggestion subunit 344 is used to make suggestions based on the user's emotions; the augmented reality unit 36 ​​includes a real-time navigation subunit 361, a real-time translation subunit 362 and an object recognition subunit 363, the real-time navigation subunit 361 is used to provide navigation reset, the real-time translation subunit 362 is used to provide translation services, and the object recognition subunit 363 is used to provide object recognition services. ;

[0069] Please refer to the attached Fig. 9 -Attached Fig.14 , an embodiment provided by the present invention: a method for implementing a smart glasses system based on adaptive vision correction and multimodal interaction, comprising step one, data collection; step two, vision detection and correction; step three, multimodal health monitoring; step four, human-computer interaction; step five, emotion recognition and management; step six, intelligent anti-fatigue; step seven, augmented reality; step eight, energy management;

[0070] In the above step 1, the data acquisition module 2 is used to collect human body data and environmental data;

[0071] In the above step 2, the data acquisition module 2 transmits the wavefront data collected by the wavefront sensor 21 to the vision detection and correction unit 31, and the vision detection and correction unit 31 automatically adjusts the refractive index of the liquid crystal variable focus lens 13. The vision detection and correction unit 31 uses the Zernike polynomial to fit the wavefront aberration, and the formula is as follows:

[0072]

[0073] Among them, ρ and θ are polar coordinates, is the Zernike polynomial, is the corresponding coefficient;

[0074] The formula for focal length is as follows:

[0075]

[0076] Where, f is the focal length, n(V) is the refractive index corresponding to the driving voltage V, and R is the radius of curvature of the lens;

[0077] In the above step 3, the data acquisition module 2 transmits the ECG signal, pulse wave signal, skin electrical signal and body temperature collected by the biosensor 22 to the multimodal health monitoring unit 32, and the multimodal health monitoring unit 32 extracts the heart rate, blood oxygen saturation and skin conductivity change rate from the ECG signal, pulse wave signal, skin electrical signal and body temperature, and then uses a multi-parameter fusion algorithm to fuse the heart rate, blood oxygen saturation, skin conductivity change rate and body temperature to obtain a feature vector, and finally inputs the feature vector into a support vector machine for health status classification;

[0078] The heart rate HR calculation formula is:

[0079]

[0080] in, is the mean RR interval;

[0081] Blood oxygen saturation SpO2 is calculated using the ratio R of red light and infrared light of the pulse wave signal:

[0082] SpO2=AB×R

[0083] Where A and B are calibration coefficients;

[0084] The calculation formula of skin conductivity change rate ΔGSR is as follows:

[0085]

[0086] Among them, GSR(t) is the skin electrical signal;

[0087] In the above step 4, the data acquisition module 2 transmits the hand image collected by the rear camera 24 and the voice signal collected by the microphone 25 to the human-computer interaction unit 33, the gesture recognition subunit 331 analyzes the hand image and recognizes the gesture command, the voice content recognition subunit 332 analyzes the voice signal and recognizes the voice command, and the command fusion subunit 333 fuses the recognized gesture and voice command to control the function of the system;

[0088] In the above step 5, the data acquisition module 2 transmits the facial image collected by the front camera 23 and the voice signal collected by the microphone 25 to the emotion recognition and management unit 34, the facial expression recognition subunit 341 analyzes the facial image and recognizes the facial expression, the voice emotion recognition subunit 342 analyzes the voice signal and recognizes the voice emotion, the emotion state determination subunit 343 performs weighted fusion recognition on the facial expression and the voice emotion and outputs the emotion recognition result, and the emotion management suggestion subunit 344 generates the emotion management suggestion according to the emotion recognition result;

[0089] In the above step 6, the data acquisition module 2 transmits the eyeball position and pupil diameter data collected by the eye movement sensor 26 to the intelligent anti-fatigue unit 35, and the intelligent anti-fatigue unit 35 extracts the blinking frequency and eyeball movement speed, determines the fatigue degree through the threshold, and adjusts the display brightness and blue light intensity according to the fatigue degree;

[0090] Blink frequency f blink The calculation formula is:

[0091]

[0092] Eye movement speed v t The calculation formula is:

[0093]

[0094] The display brightness L is calculated as follows:

[0095] L=L0×(1-αF)

[0096] The calculation formula of blue light intensity B is:

[0097] B=B0×(1-βF)

[0098] Among them, α and β are adjustment coefficients;

[0099] In the above step 7, the data acquisition module 2 transmits the image data collected by the rear camera 24 to the augmented reality unit 36, the real-time navigation subunit 361 analyzes the image data and updates the three-dimensional point cloud map, the real-time translation subunit 362 analyzes the image data, identifies the text area, and outputs the translation result, and the object recognition subunit 363 analyzes the image data, identifies the object, and outputs the object category and position boundary box;

[0100] In the above step eight, the data acquisition module 2 transmits the light intensity data collected by the light sensor 27 to the energy management module 37, and the energy management module 37 predicts the output P of the solar power supply module 12. solar (t), and monitor the power consumption P of each module in real time i (t), and use optimization algorithms to dynamically adjust the energy supply of each module to solve the following problems:

[0101]

[0102] The constraints are:

[0103] P total (t)≤P solar (t)+P battery (t).

[0104] Among them, Ptotal (t) is the total energy consumption, P battery (t) is the electric energy stored in the energy storage module 14. The charging and discharging strategy adopted by the energy management module 37 is:

[0105] When P solar (t)>P total At (t), the solar power supply module 12 charges the energy storage module 14;

[0106] When P solar (t)<P total (t), the energy storage module 14 supplies power.

[0107] Based on the above, the advantage of the present invention is that when the present invention is used, the data acquisition module 2 is used to collect human body data and environmental data, specifically: the wavefront sensor 21 is used to collect wavefront data, the electrocardiogram sensor 221 is used to collect electrocardiogram signals, the photoelectric volume pulse wave sensor 222 is used to collect pulse wave signals, the electrocardiogram signal skin electricity sensor 223 is used to collect skin electricity signals, the temperature sensor 224 is used to collect temperature, the front camera 23 is used to collect facial image information, the rear camera 24 is used to collect environmental image information, the microphone 25 is used to collect user voice signals, the eye movement sensor 26 is used to collect eyeball position and pupil diameter data, and the light sensor 27 is used to collect light intensity data; the wavefront sensor 21 collects The wavefront data is transmitted to the vision detection and correction unit 31, which first performs an aberration analysis on the wavefront data, and then extracts low-order aberrations related to refractive errors from the fitting coefficients, specifically including spherical aberration and cylindrical aberration, and then calculates the refractive power based on the extracted low-order aberrations, and then generates the voltage required for adjusting the liquid crystal variable focus lens 13 based on the refractive power, thereby adjusting the refractive index of the liquid crystal variable focus lens 13; the electrocardiogram signal, pulse wave signal, skin electrical signal and body temperature collected by the biosensor 22 are transmitted to the multimodal health monitoring unit 32, and the multimodal health monitoring unit 32 filters and normalizes the signal, and then performs feature extraction. The extracted feature data specifically includes heart rate, blood oxygen saturation and skin conductivity. The hand image captured by the rear camera 24 and the voice signal captured by the microphone 25 are transmitted to the human-computer interaction unit 33. The gesture recognition subunit 331 performs background subtraction and normalization on the hand image, and then uses a three-dimensional convolutional neural network to extract spatial and temporal features, uses a long short-term memory network to perform time series modeling, outputs gesture category probabilities, and realizes the recognition of gesture commands. The voice content recognition subunit 332 calculates the Mel frequency cepstral coefficients of the voice signal, and then uses the Transformer model to convert the voice into text, outputs text commands, and realizes the recognition of voice commands. Finally, the command fusion subunit 333 The recognized gestures and voice commands are integrated to control the functions of the system; the facial images collected by the front camera 23 and the voice signals collected by the microphone 25 are transmitted to the emotion recognition and management unit 34, the facial expression recognition subunit 341 obtains the facial image, performs face detection and key point positioning, then calculates the facial action unit, and then uses the convolutional neural network to classify the expression; the voice emotion recognition subunit 342 extracts the fundamental frequency, energy and resonance peak characteristics of the voice, and uses the long short-term memory network combined with time series information to recognize the voice emotion; the emotional state determination subunit 343 performs weighted fusion recognition of the expression and the voice emotion, and outputs the emotion recognition result; the emotion management suggestion subunit 344 generates emotion management suggestions based on the emotion recognition result;The eye position and pupil diameter data collected by the eye movement sensor 26 are transmitted to the intelligent anti-fatigue unit 35, which extracts the blinking frequency and eye movement speed, determines the fatigue level through a threshold, and adjusts the display brightness and blue light intensity according to the fatigue level; the image data collected by the rear camera 24 is transmitted to the augmented reality unit 36, the real-time navigation subunit 361 extracts the image feature points, and then matches the feature points of the current frame with the key frame through the feature descriptor, and then uses the least squares method to calculate the position and posture of the rear camera 24, and updates the three-dimensional point cloud map accordingly; the real-time translation subunit 362 uses a deep learning model to detect the text area, and then applies optical character recognition The technology recognizes the text, and then uses the neural machine translation model to translate the source language into the target language, and finally superimposes the translation result in the form of AR in the text area; the object recognition subunit 363 uses the YOLOv5 model to recognize the object and output the object category and position boundary box; the light intensity data collected by the light sensor 27 is transmitted to the energy management module 37, and the energy management module 37 predicts the output of the solar power supply module 12 and monitors the power consumption of each module in real time. At the same time, the optimization algorithm is used to dynamically adjust the energy supply of each module and automatically adjust the charging and discharging strategy of the energy storage module 14; wherein, the speaker 11 is used to play voice, and the frame 1 is the main frame part of the smart glasses. ;

[0108] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A smart glasses system based on adaptive vision correction and multimodal interaction, comprising a glasses frame (1), a speaker (11), a solar power supply module (12), a liquid crystal variable focus lens (13), an energy storage module (14), a data acquisition module (2) and a data processing module (3), characterized in that: A data processing module (3) is installed on the frame (1); the data processing module (3) is electrically connected to a speaker (11), a solar power supply module (12), a liquid crystal variable focus lens (13) and a data acquisition module (2); and the speaker (11), the solar power supply module (12), the liquid crystal variable focus lens (13) and the data acquisition module (2) are all installed on the frame (1).

2. The smart glasses system based on adaptive vision correction and multimodal interaction according to claim 1, characterized in that: The data acquisition module (2) comprises a wavefront sensor (21), a biosensor (22), a front camera (23), a rear camera (24), a microphone (25), an eye movement sensor (26) and a light sensor (27); the biosensor (22) comprises an electrocardiogram sensor (221), a photoelectric volumetric pulse wave sensor (222), a skin electricity sensor (223) and a temperature sensor (224).

3. The smart glasses system based on adaptive vision correction and multimodal interaction according to claim 1, characterized in that: The data processing module (3) comprises a vision detection and correction unit (31), a multimodal health monitoring unit (32), a human-computer interaction unit (33), an emotion recognition and management unit (34), an intelligent anti-fatigue unit (35), an augmented reality unit (36) and an energy management module (37).

4. The smart glasses system based on adaptive vision correction and multimodal interaction according to claim 3, characterized in that: The human-computer interaction unit (33) includes a gesture recognition subunit (331), a voice content recognition subunit (332) and a command fusion subunit (333), and the emotion recognition and management unit (34) includes a facial expression recognition subunit (341), a voice emotion recognition subunit (342), an emotion state determination subunit (343) and an emotion management suggestion subunit (344).

5. The smart glasses system based on adaptive vision correction and multimodal interaction according to claim 3, characterized in that: The augmented reality unit (36) includes a real-time navigation subunit (361), a real-time translation subunit (362) and an object recognition subunit (363).

6. A method for implementing a smart glasses system based on adaptive vision correction and multimodal interaction, comprising step 1, data collection; step 2, vision detection and correction; step 3, multimodal health monitoring; step 4, human-computer interaction; step 5, emotion recognition and management; step 6, intelligent anti-fatigue; step 7, augmented reality; step 8, energy management; characterized in that: In the above step 1, the data collection module (2) is used to collect human body data and environmental data; In the above step 2, the data acquisition module (2) transmits the wavefront data collected by the wavefront sensor (21) to the vision detection and correction unit (31), and the vision detection and correction unit (31) automatically adjusts the refractive index of the liquid crystal variable focus lens (13); In the above step 3, the data acquisition module (2) transmits the electrocardiogram signal, pulse wave signal, skin electrical signal and body temperature collected by the biosensor (22) to the multimodal health monitoring unit (32), and the multimodal health monitoring unit (32) automatically outputs the health status assessment result; In the above step 4, the data acquisition module (2) transmits the hand image collected by the rear camera (24) and the voice signal collected by the microphone (25) to the human-computer interaction unit (33), the gesture recognition subunit (331) analyzes the hand image and recognizes the gesture command, the voice content recognition subunit (332) analyzes the voice signal and recognizes the voice command, and the command fusion subunit (333) fuses the recognized gesture and voice command to control the function of the system; In the above step 5, the data acquisition module (2) transmits the facial image collected by the front camera (23) and the voice signal collected by the microphone (25) to the emotion recognition and management unit (34), the facial expression recognition subunit (341) analyzes the facial image and recognizes the facial expression, the voice emotion recognition subunit (342) analyzes the voice signal and recognizes the voice emotion, the emotion state determination subunit (343) performs weighted fusion recognition on the facial expression and the voice emotion and outputs the emotion recognition result, and the emotion management suggestion subunit (344) generates emotion management suggestions based on the emotion recognition result; In the above step 6, the data acquisition module (2) transmits the eyeball position and pupil diameter data collected by the eye movement sensor (26) to the intelligent anti-fatigue unit (35), and the intelligent anti-fatigue unit (35) determines the degree of fatigue and adjusts the display brightness and blue light intensity according to the degree of fatigue; In the above step 7, the data acquisition module (2) transmits the image data collected by the rear camera (24) to the augmented reality unit (36), the real-time navigation subunit (361) analyzes the image data and updates the three-dimensional point cloud map, the real-time translation subunit (362) analyzes the image data, identifies the text area, and outputs the translation result, and the object recognition subunit (363) analyzes the image data, identifies the object, and outputs the object category and position boundary box; In the above step eight, the data acquisition module (2) transmits the light intensity data collected by the light sensor (27) to the energy management module (37), and the energy management module (37) predicts the output P of the solar power supply module (12). solar (t), and monitor the power consumption P of each module in real time i (t), and use optimization algorithms to dynamically adjust the energy supply of each module to solve the following problems: The constraints are: P total (t)≤P solar (t)+P battery (t)。 Among them, P total (t) is the total energy consumption, P battery (t) is the electric energy stored in the energy storage module (14).

7. The method for implementing the smart glasses system based on adaptive vision correction and multimodal interaction according to claim 6, characterized in that: In the step 2, the vision detection and correction unit (31) uses Zernike polynomials to perform fitting calculation on the wavefront aberration, and the formula is as follows: Among them, ρ and θ are polar coordinates, is the Zernike polynomial, is the corresponding coefficient; The formula for focal length is as follows: Wherein, f is the focal length, n(V) is the refractive index corresponding to the driving voltage V, and R is the radius of curvature of the lens.

8. The method for implementing the smart glasses system based on adaptive vision correction and multimodal interaction according to claim 6, characterized in that: In the step 3, the multimodal health monitoring unit (32) extracts the heart rate, blood oxygen saturation and skin conductivity change rate from the electrocardiogram signal, the pulse wave signal, the skin electrical signal and the body temperature, and then fuses the heart rate, blood oxygen saturation, skin conductivity change rate and body temperature using a multi-parameter fusion algorithm to obtain a feature vector, and finally inputs the feature vector into a support vector machine to classify the health status; The heart rate HR calculation formula is: in, is the mean RR interval; Blood oxygen saturation SpO2 is calculated using the ratio R of red light and infrared light of the pulse wave signal: SpO2=AB×R Where A and B are calibration coefficients; The calculation formula of skin conductivity change rate ΔGSR is as follows: Among them, GSR(t) is the skin electrical signal.

9. The method for implementing the smart glasses system based on adaptive vision correction and multimodal interaction according to claim 6, characterized in that: In the step six, the intelligent anti-fatigue unit (35) extracts the blinking frequency and the eye movement speed, and determines the fatigue degree through a threshold value; Blink frequency f blink The calculation formula is: Eye movement speed v t The calculation formula is: The display brightness L is calculated as follows: L=L0×(1-αF) The calculation formula of blue light intensity B is: B=B0×(1-βF) Among them, α and β are adjustment coefficients.

10. The method for implementing the smart glasses system based on adaptive vision correction and multimodal interaction according to claim 6, characterized in that: In step eight, the charging and discharging strategy adopted by the energy management module (37) is: When P solar (t)>P total At (t), the solar power supply module (12) charges the energy storage module (14); When P solar (t)<P total (t), the energy storage module (14) supplies power.

Images