Intelligent glasses system based on machine vision and voice interaction

By integrating machine vision and large language models in smart glasses, real-time recognition and voice broadcast of blind users' environmental information is achieved, which solves the problem of inconvenience in the blind and provides more convenient navigation and environmental exploration capabilities.

CN118897625APending Publication Date: 2024-11-05CHINA YUANCHEN HOLDINGS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411050521.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

Existing smart glasses products fail to effectively combine the visual or image processing capabilities of large language models, resulting in inconvenience for blind people to travel, mainly relying on guide dogs or blind crutches.

Method used

Design a smart glasses system based on machine vision and voice interaction, including vision acquisition module, sound acquisition module, machine vision analysis module and large language analysis module. By identifying environmental obstacles and navigation information in real time, voice broadcasts are generated to assist blind users navigation.

Benefits of technology

It significantly improves the travel convenience of blind users, and replaces traditional crutches through environmental information identification and intelligent interaction, providing more convenient environmental exploration and navigation support.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897625A_ABST
    Figure CN118897625A_ABST
Patent Text Reader

Abstract

The invention discloses an intelligent glasses system based on machine vision and voice interaction, and belongs to the technical field of intelligent glasses, environmental information of a blind user can be acquired through a vision acquisition module, recognition of the environmental information can be realized through a machine vision analysis module, and finally broadcasting can be performed through a voice interaction module. The walking stick is used for replacing a walking stick of a blind user to explore the environment, the traveling convenience of the blind user can be greatly improved, the voice data collected by the voice collection module is analyzed through the big language analysis module, dialogues or instructions are automatically recognized, intelligent interaction is formed, and the blind user can use the walking stick more conveniently.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of smart glasses, and in particular relates to a smart glasses system based on machine vision and voice interaction. Background Art

[0002] With the development of science and technology, artificial intelligence technology has been widely used in various fields. Among them, smart glasses, as an emerging smart wearable device, have received more and more attention. At present, the smart glasses products available on the market mainly have functions such as video shooting, real-time transmission and image recognition, but have not yet combined with the visual or image processing capabilities of large language models, and there are few smart glasses products designed specifically for the blind. In the existing technology, the blind mainly rely on guide dogs or canes, which makes travel extremely inconvenient. Summary of the invention

[0003] The present invention provides a smart glasses system based on machine vision and voice interaction, which is used to solve the problem in the prior art that blind people mainly rely on guide dogs or blind people's crutches to travel, which causes great inconvenience.

[0004] A smart glasses system based on machine vision and voice interaction, comprising: a visual acquisition module, a sound acquisition module, a machine vision analysis module, a large language analysis module and a voice interaction module;

[0005] The visual acquisition module is used to collect video stream data in front of the smart glasses and transmit the video stream data to the machine vision analysis module;

[0006] The sound collection module is used to collect the sound data spoken by the blind user wearing the smart glasses, and transmit the sound data to the large language analysis module;

[0007] The machine vision analysis module is used to analyze the video stream data transmitted by the visual acquisition module, identify obstacle information in front of the smart glasses in real time, and transmit the obstacle information to the voice interaction module;

[0008] The large language analysis module is used to generate voice interaction data based on the sound data using a large language model, and transmit the voice interaction data to the voice interaction module;

[0009] The voice interaction module is used to broadcast the obstacles identified by the machine vision analysis module and to broadcast the voice interaction data generated by the large language analysis module.

[0010] Furthermore, the visual acquisition module is also used to: when the blind user is navigating, identify the blind path in real time and generate navigation guidance data to guide the blind user to walk on the blind path, so that the voice interaction module broadcasts the navigation guidance data, thereby realizing navigation for the blind user.

[0011] Furthermore, the machine vision analysis module is also used to: analyze the video stream data transmitted by the visual acquisition module, identify the object information in front of the smart glasses in real time, and broadcast the object information to the blind user.

[0012] Furthermore, analyzing the video stream data transmitted by the visual acquisition module to identify obstacles in front of the smart glasses in real time includes:

[0013] Based on a preset data sampling frequency, the video stream data transmitted by the visual acquisition module is subjected to frame extraction processing to obtain a video key frame;

[0014] Using a pre-trained machine vision model to identify the video key frames, and obtain prediction boxes and prediction categories of each object in the video key frames;

[0015] Determine whether the predicted box of the object in the key frame of the video is within the restricted area. If so, the predicted category of the object is used as an obstacle directly in front of the smart glasses. Otherwise, the predicted category of the object is not used as an obstacle directly in front of the smart glasses.

[0016] The predicted frame of each object in the key frame of the video is judged to obtain the obstacles directly in front of the smart glasses.

[0017] Furthermore, after real-time identification of obstacles in front of the smart glasses, the following steps are also included:

[0018] Identify the distance and angle between the obstacle in front of the smart glasses and the smart glasses, and input the distance and angle between the obstacle and the smart glasses into the voice interaction module for broadcasting;

[0019] The angle is used to represent the angle between the line between the smart glasses and the obstacle and the base line, with the front of the smart glasses as the base line.

[0020] Further, the video stream data transmitted by the visual acquisition module is analyzed and frame-extracted to obtain video key frames, including:

[0021] Based on the preset data sampling frequency, the real-time transmitted video stream data is subjected to frame extraction processing to obtain video key frames.

[0022] Furthermore, the video key frames are identified using a pre-trained machine vision model to obtain prediction boxes and prediction categories of each object in the video key frames, including:

[0023] A machine vision model is constructed by using a YOLO model, and the machine vision model is trained by using an adaptive optimization method to obtain a pre-trained machine vision model;

[0024] The video key frames are used as input to the pre-trained machine vision model to obtain the predicted boxes and predicted categories of each object in the video key frames.

[0025] Furthermore, the machine vision model is trained using an adaptive optimization method to obtain a pre-trained machine vision model, including:

[0026] A1. Initializing the hyperparameters of the machine vision model to obtain multiple hyperparameter encodings;

[0027] A2. Obtain the loss function value corresponding to each hyperparameter encoding, and select the hyperparameter encoding with the smallest loss function as the optimal encoding;

[0028] A3. Adopting adaptive search step size to perform adaptive multi-coding guided optimization on hyperparameter coding to increase population diversity and obtain hyperparameter coding after initial optimization;

[0029] A4, using an adaptive search step size, and adaptively adjusting and optimizing the hyperparameter encoding after the initial optimization according to the optimal encoding, to obtain the hyperparameter encoding after the secondary optimization;

[0030] A5. Perform local optimal development optimization on the hyperparameter encoding after the secondary optimization according to the optimal encoding to obtain the hyperparameter encoding after the tertiary optimization;

[0031] A6. Re-obtain the optimal encoding based on the hyperparameter encoding after three optimizations;

[0032] A7. Repeat steps A3 to A6 until the number of training times reaches the maximum number of training times, and use the current optimal encoding as the final hyperparameter of the machine vision model to obtain a pre-trained machine vision model.

[0033] Further, determining whether the prediction box of the object in the key frame of the video is located in the restricted area includes:

[0034] Determine the center point of the predicted box of the object in the video key frame to obtain the target center point;

[0035] Determine the vertical center line and the horizontal center line of the video key frame;

[0036] Determine whether the distance between the target center point and the vertical center line is less than a first distance threshold, if so, determine that the prediction box of the object in the video key frame is within the restricted area, otherwise further determine the relationship between the horizontal center line and the target center point;

[0037] Determine whether the distance between the target center point and the vertical cross center line is less than the second distance threshold. If so, determine that the prediction box of the object in the video key frame is located in the restricted area; otherwise, determine that the prediction box of the object in the video key frame is not located in the restricted area.

[0038] Furthermore, based on the sound data, a large language model is used to generate voice interaction data, including:

[0039] Recognize the sound data collected by the sound collection module to obtain target text information;

[0040] The target text information is used as input of a large language model to generate voice interaction data.

[0041] Furthermore, the large language analysis module is also used for:

[0042] Recognize the sound data collected by the sound collection module to obtain target text information;

[0043] The target text information is matched with the pre-stored control instructions by similarity, and when there is a control instruction with a similarity matching degree greater than a preset threshold, the control instruction with the greatest similarity is used as the target instruction;

[0044] The target instruction is transmitted to the smart glasses so that the smart glasses perform corresponding operations.

[0045] The present invention provides a smart glasses system based on machine vision and voice interaction. The visual collection module can collect environmental information of blind users, and the machine vision analysis module can recognize the environmental information. Finally, the voice interaction module can broadcast the information to replace the blind user's crutch for environmental exploration, which can greatly improve the travel convenience of the blind user. The large language analysis module analyzes the sound data collected by the sound collection module, and automatically recognizes dialogues or commands to form intelligent interaction, which is more convenient for blind users to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0047] Figure 1 A schematic diagram of the structure of a smart glasses system based on machine vision and voice interaction provided by an embodiment of the present invention.

[0048] Figure 2 A schematic diagram of a restricted area provided in an embodiment of the present invention.

[0049] Among them, 101-visual acquisition module, 102-sound acquisition module, 103-machine vision analysis module, 104-large language analysis module, 105-voice interaction module.

[0050] The above drawings have shown clear embodiments of the present invention, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0052] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 As shown, an embodiment of the present invention provides a smart glasses system based on machine vision and voice interaction, including: a vision acquisition module 101, a sound acquisition module 102, a machine vision analysis module 103, a large language analysis module 104 and a voice interaction module 105.

[0054] The visual acquisition module 101 is used to acquire video stream data in front of the smart glasses and transmit the video stream data to the machine vision analysis module.

[0055] The sound collection module 102 is used to collect the sound data spoken by the blind user wearing the smart glasses, and transmit the sound data to the large language analysis module.

[0056] The machine vision analysis module 103 is used to analyze the video stream data transmitted by the visual acquisition module, identify obstacle information in front of the smart glasses in real time, and transmit the obstacle information to the voice interaction module.

[0057] The large language analysis module 104 is used to generate voice interaction data based on the sound data using a large language model, and transmit the voice interaction data to the voice interaction module.

[0058] The voice interaction module 105 is used to announce obstacles identified by the machine vision analysis module and to announce voice interaction data generated by the large language analysis module.

[0059] The present invention provides a smart glasses system based on machine vision and voice interaction. The visual collection module can collect environmental information of blind users, and the machine vision analysis module can recognize the environmental information. Finally, the voice interaction module can broadcast the information to replace the blind user's crutch for environmental exploration, which can greatly improve the travel convenience of the blind user. The large language analysis module analyzes the sound data collected by the sound collection module, and automatically recognizes dialogues or commands to form intelligent interaction, which is more convenient for blind users to use.

[0060] In an embodiment of the present invention, the visual acquisition module is also used to: when a blind user is navigating, identify the blind path in real time, and generate navigation guidance data to guide the blind user to walk on the blind path, so that the voice interaction module broadcasts the navigation guidance data, thereby realizing navigation for the blind user.

[0061] In an embodiment of the present invention, the machine vision analysis module is further used to analyze the video stream data transmitted by the visual acquisition module, identify the object information in front of the smart glasses in real time, and broadcast the object information to the blind user. It is worth noting that the object information can also be identified in the same way as the obstacle.

[0062] Navigation is performed through smart glasses, and the blind user's location and route are determined through machine vision technology and positioning information, so that navigation guidance data can be generated. Combined with obstacle identification, it greatly facilitates the travel of blind users.

[0063] Optionally, in addition to identifying obstacles, zebra crossings and red lights can also be identified to guide blind users to cross the road safely.

[0064] In an embodiment of the present invention, analyzing the video stream data transmitted by the visual acquisition module to identify obstacles in front of the smart glasses in real time includes:

[0065] Based on a preset data sampling frequency, the video stream data transmitted by the visual acquisition module is subjected to frame extraction processing to obtain a video key frame;

[0066] Taking into account that blind users move relatively slowly, the data sampling frequency can be preset, video key frames can be extracted at the data sampling frequency, and the video key frames can be recognized in real time to achieve visual interaction.

[0067] Using a pre-trained machine vision model to identify the video key frames, and obtain prediction boxes and prediction categories of each object in the video key frames;

[0068] The target tracking model can be used as a machine vision model and hyperparameters can be optimized in advance, so that targets in video key frames can be identified in real time.

[0069] It is worth noting that if more perfect functions are needed, the machine vision model needs to be modified a lot so that the machine vision model can recognize more objects and issue more prompt information for different objects, making it more convenient for blind users to travel. For example, the camera coordinate system can be converted to the world coordinate system to determine the position of the obstacle, so it can be reported that there is an obstacle in front, and the specific position is 45 degrees to the left and 3 meters away.

[0070] Determine whether the predicted box of the object in the key frame of the video is within the restricted area. If so, the predicted category of the object is used as an obstacle directly in front of the smart glasses. Otherwise, the predicted category of the object is not used as an obstacle directly in front of the smart glasses.

[0071] The predicted frame of each object in the key frame of the video is judged to obtain the obstacles directly in front of the smart glasses.

[0072] The size of the restricted area can be set according to the camera's field of view width to ensure that the front range of the blind user is safe, while reducing information interference from other areas to avoid confusing information interfering with the blind user's judgment.

[0073] In the embodiment of the present invention, after real-time identification of obstacles in front of the smart glasses, the method further includes:

[0074] Identify the distance and angle between the obstacle in front of the smart glasses and the smart glasses, and input the distance and angle between the obstacle and the smart glasses into the voice interaction module for broadcasting;

[0075] The angle is used to represent the angle between the line between the smart glasses and the obstacle and the base line, with the front of the smart glasses as the base line (i.e., the direction perpendicular to the shooting screen).

[0076] For example, the camera coordinate system can be converted into the world coordinate system to determine the coordinate points of different objects and finally determine the connection lines between different objects. In order to further improve the coordinate acquisition capability, a depth camera can be set or binocular vision can be used to determine the coordinates to determine the location of obstacles.

[0077] In an embodiment of the present invention, analyzing and performing frame extraction processing on the video stream data transmitted by the visual acquisition module to obtain video key frames includes:

[0078] Based on the preset data sampling frequency, the real-time transmitted video stream data is subjected to frame extraction processing to obtain video key frames.

[0079] In an embodiment of the present invention, a pre-trained machine vision model is used to identify the video key frame to obtain a predicted frame and a predicted category of each object in the video key frame, including:

[0080] A machine vision model is constructed by using a YOLO model, and the machine vision model is trained by using an adaptive optimization method to obtain a pre-trained machine vision model;

[0081] YOLO (You Only Look Once) is a popular computer vision recognition algorithm used to implement object detection tasks. It is characterized by high speed and can detect objects in images in near real time. In order to better identify obstacles, the embodiment of the present invention preferably uses YOLOv4 or YOLOv5.

[0082] The video key frames are used as input to the pre-trained machine vision model to obtain the predicted boxes and predicted categories of each object in the video key frames.

[0083] In an embodiment of the present invention, the machine vision model is trained using an adaptive optimization method to obtain a pre-trained machine vision model, including:

[0084] A1. Initializing the hyperparameters of the machine vision model to obtain multiple hyperparameter encodings;

[0085] A2. Obtain the loss function value corresponding to each hyperparameter encoding, and select the hyperparameter encoding with the smallest loss function as the optimal encoding;

[0086] A3. Adopting adaptive search step size to perform adaptive multi-coding guided optimization on hyperparameter coding to increase population diversity and obtain hyperparameter coding after initial optimization;

[0087] Generally speaking, the adaptive step size of the existing technology changes in a linear decreasing manner. Although it can improve the global search capability of the algorithm in the early stage, it greatly reduces the traversability and flexibility of the population, making the algorithm more likely to fall into the local optimum.

[0088] Therefore, an embodiment of the present invention provides an adaptive search step size of:

[0089] μ=0.8*(1+θ)*exp((Tt) / T) 2

[0090] Among them, μ represents the adaptive search step size, θ represents the intermediate parameter, and the value of the intermediate parameter is: generate a random number rand1 between (0,1), when the random number rand1 is greater than 0.5, then θ = 2(1-rand1), otherwise θ = 2rand1; t represents the current number of training times, and T represents the maximum number of training times.

[0091] The modified adaptive search step size shows a downward trend during the entire training process, and there are fluctuations in the area, thus forming a sawtooth change curve with an overall downward trend, which can enhance the traversal, randomness and flexibility of individual searches, and enhance the global exploration ability of the population in the early stage of the algorithm. In addition, the fluctuation range tends to become narrower and narrower as the number of searches increases, and the step size gradually decreases, which improves the local search ability of the algorithm as the algorithm search time increases, thereby enhancing the local development ability of the algorithm.

[0092] Optionally, an adaptive multi-encoding guided optimization is performed on the hyperparameter encoding using an adaptive search step size:

[0093] x i (t+1)=x i (t)+rand2*θ*[(0.7x G (t)+0.3x R (t))-x i (t)]

[0094] Among them, x i (t) represents the i-th hyperparameter encoding in the t-th training, i = 1, 2, ..., M, M represents the total number of hyperparameter encodings, x i (t+1) represents the hyperparameter encoding x after the initial optimization i (t), rand2 represents a uniform random number in [0,1], x G (t) represents the optimal coding, x R (t) represents the same as x G (t) and x i (t) Different random codes,

[0095] The embodiment of the present invention adopts the weighted method of optimal coding and random coding to guide optimization, which increases disturbance while ensuring that the moving direction is biased towards the current global optimal position, increases the diversity of the algorithm, and avoids the situation where the algorithm is premature and falls into the local optimum. By citing the adaptive search step size, the search distance is more diverse, the flexibility and maneuverability of the algorithm solution are improved, and the ability to jump out of the local optimum is enhanced.

[0096] A4, using an adaptive search step size, and adaptively adjusting and optimizing the hyperparameter encoding after the initial optimization according to the optimal encoding, to obtain the hyperparameter encoding after the secondary optimization;

[0097] Optionally, the hyperparameter encoding after the initial optimization is adaptively adjusted and optimized according to the optimal encoding:

[0098] x n (t+1)=x n (t)+γ*exp(|ξ n |)*μ*x n (t)+c*μ*(x G (t)-x n (t))

[0099] Among them, x n (t) represents the hyperparameter encoding after the nth initial optimization, n = 1, 2, ..., M, γ represents the symbolic parameter, γ = ξ n / |ξ n |,ξ n Indicates the quality of the search direction. f n (t) represents the fitness value of the hyperparameter encoding in the tth training after the nth initial optimization, f n (t-1) represents the fitness value of the hyperparameter encoding after the nth initial optimization in the t-1th training, and the fitness value is 1 / (loss function value + 0.001); μ represents the adaptive search step size, c represents the learning factor (can be set to 0.01), and x G (t) represents the optimal coding.

[0100] When the quality of the search direction ξ n When it is less than 0, it means that the last update is far from the optimal position, and this update should be corrected. Then, by correcting the weights and guiding the optimal hyperparameter encoding, a large step search can be performed when the hyperparameter encoding is in a bad position; a small step search can be performed when it is in a good position, which can not only increase the diversity of the search, but also prevent the algorithm from falling into the local optimum. When reaching the final stage, you can continue to search near the optimal position to increase the convergence accuracy.

[0101] A5. Perform local optimal development optimization on the hyperparameter encoding after the secondary optimization according to the optimal encoding to obtain the hyperparameter encoding after the tertiary optimization;

[0102] Optionally, the hyperparameter encoding after the secondary optimization is optimized to be locally optimal according to the optimal encoding:

[0103] x m (t+1)=x G (t)+d*rand m *μ*x G (t)

[0104] Among them, d represents a constant term, and its value is 0.7; rand m Represented as the hyperparameter encoding x after the mth secondary optimization m (t) generates a random number, and rand m is a uniform random number in [-1,1], m=1,2,…,M, x m (t+1) represents the hyperparameter encoding x after three optimizations m (t).

[0105] It is worth noting that although the above local optimal development optimization can greatly improve the algorithm convergence accuracy and local development ability, it increases the possibility of falling into the local optimum. Therefore, the local optimal development optimization can be performed after the algorithm is executed for a certain number of times. For example, after the algorithm has executed 0.9T, the local optimal development optimization is performed. In order to ensure the correctness of the algorithm, out-of-bounds processing can also be performed after each hyperparameter encoding is changed.

[0106] Through the above-mentioned local optimal development and optimization, the convergence ability of the algorithm can be greatly improved. Combined with the aforementioned global development, the problem that the existing technology is prone to falling into local optimality is avoided, and the ability to recognize targets in images is ultimately improved, thereby better assisting blind users in activities.

[0107] A6. Re-obtain the optimal encoding based on the hyperparameter encoding after three optimizations;

[0108] A7. Repeat steps A3 to A6 until the number of training times reaches the maximum number of training times, and use the current optimal encoding as the final hyperparameter of the machine vision model to obtain a pre-trained machine vision model.

[0109] In an embodiment of the present invention, determining whether a prediction box of an object in a key frame of a video is located in a restricted area includes:

[0110] Determine the center point of the predicted box of the object in the video key frame to obtain the target center point;

[0111] Determine the vertical center line and the horizontal center line of the video key frame;

[0112] Determine whether the distance between the target center point and the vertical center line is less than a first distance threshold, if so, determine that the prediction box of the object in the video key frame is within the restricted area, otherwise further determine the relationship between the horizontal center line and the target center point;

[0113] Determine whether the distance between the target center point and the vertical cross center line is less than the second distance threshold. If so, determine that the prediction box of the object in the video key frame is located in the restricted area; otherwise, determine that the prediction box of the object in the video key frame is not located in the restricted area.

[0114] like Figure 2 As shown, an example diagram of a restricted area is provided in an embodiment of the present invention to facilitate those skilled in the art to better understand the technical solution described in the embodiment of the present invention. It is worth noting that the four sides of the restricted area can be adjusted and the distance from the center line can be different to facilitate the staff to adjust it to a state suitable for blind users to travel.

[0115] In an embodiment of the present invention, based on the sound data, a large language model is used to generate voice interaction data, including:

[0116] Recognize the sound data collected by the sound collection module to obtain target text information;

[0117] The target text information is used as input of a large language model to generate voice interaction data.

[0118] In the embodiment of the present invention, the large language analysis module is further used to:

[0119] Recognize the sound data collected by the sound collection module to obtain target text information;

[0120] The target text information is matched with the pre-stored control instructions by similarity, and when there is a control instruction with a similarity matching degree greater than a preset threshold, the control instruction with the greatest similarity is used as the target instruction;

[0121] The target instruction is transmitted to the smart glasses so that the smart glasses perform corresponding operations. For example, if the weather report instruction is given, the smart glasses obtain the weather information and report it.

[0122] Big language models are AI-based algorithms that can understand and generate human language. These models are usually deep learning neural networks that are trained with large amounts of text data and are able to learn the structure and grammatical rules of a language. A typical application of big language models is natural language processing (NLP), such as machine translation, sentiment analysis, text summarization, question-answering systems, etc. These models perform well when processing large amounts of text data and are able to identify and understand complex patterns in language. As the size and complexity of the models increase, big language models begin to have more advanced cognitive capabilities, such as generating coherent text, understanding implicit meanings, making inferences, and completing complex language tasks.

[0123] Therefore, the embodiment of the present invention uses a large language model to recognize text information obtained by voice conversion, so as to determine the user's intention, analyze the intention, and generate dialogue or operation instructions.

[0124] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0129] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A smart glasses system based on machine vision and voice interaction, characterized in that: include: Visual acquisition module, sound acquisition module, machine vision analysis module, large language analysis module and voice interaction module; The visual acquisition module is used to collect video stream data in front of the smart glasses and transmit the video stream data to the machine vision analysis module; The sound collection module is used to collect the sound data spoken by the blind user wearing the smart glasses, and transmit the sound data to the large language analysis module; The machine vision analysis module is used to analyze the video stream data transmitted by the visual acquisition module, identify obstacle information in front of the smart glasses in real time, and transmit the obstacle information to the voice interaction module; The large language analysis module is used to generate voice interaction data based on the sound data using a large language model, and transmit the voice interaction data to the voice interaction module; The voice interaction module is used to broadcast the obstacles identified by the machine vision analysis module and to broadcast the voice interaction data generated by the large language analysis module.

2. The smart glasses system based on machine vision and voice interaction according to claim 1, characterized in that: The visual acquisition module is also used to: when the blind user is navigating, identify the blind path in real time and generate navigation guidance data to guide the blind user to walk on the blind path, so that the voice interaction module broadcasts the navigation guidance data, thereby realizing navigation for the blind user.

3. The smart glasses system based on machine vision and voice interaction according to claim 1, characterized in that: The machine vision analysis module is also used to analyze the video stream data transmitted by the visual acquisition module, identify the object information in front of the smart glasses in real time, and broadcast the object information to the blind user.

4. The smart glasses system based on machine vision and voice interaction according to claim 1, characterized in that: Analyze the video stream data transmitted by the visual acquisition module to identify obstacles in front of the smart glasses in real time, including: Based on a preset data sampling frequency, the video stream data transmitted by the visual acquisition module is subjected to frame extraction processing to obtain a video key frame; Using a pre-trained machine vision model to identify the video key frames, and obtain prediction boxes and prediction categories of each object in the video key frames; Determine whether the predicted box of the object in the key frame of the video is within the restricted area. If so, the predicted category of the object is used as an obstacle directly in front of the smart glasses. Otherwise, the predicted category of the object is not used as an obstacle directly in front of the smart glasses. The predicted frame of each object in the key frame of the video is judged to obtain the obstacles directly in front of the smart glasses.

5. The smart glasses system based on machine vision and voice interaction according to claim 4, characterized in that: After real-time identification of obstacles in front of the smart glasses, it also includes: Identify the distance and angle between the obstacle in front of the smart glasses and the smart glasses, and input the distance and angle between the obstacle and the smart glasses into the voice interaction module for broadcasting; The angle is used to represent the angle between the line between the smart glasses and the obstacle and the base line, with the front of the smart glasses as the base line.

6. The smart glasses system based on machine vision and voice interaction according to claim 5, characterized in that: The video key frames are identified using a pre-trained machine vision model to obtain prediction boxes and prediction categories of each object in the video key frames, including: A machine vision model is constructed by using a YOLO model, and the machine vision model is trained by using an adaptive optimization method to obtain a pre-trained machine vision model; The video key frames are used as input to the pre-trained machine vision model to obtain the predicted boxes and predicted categories of each object in the video key frames.

7. The smart glasses system based on machine vision and voice interaction according to claim 6, characterized in that: The machine vision model is trained using an adaptive optimization method to obtain a pre-trained machine vision model, including: A1. Initializing the hyperparameters of the machine vision model to obtain multiple hyperparameter encodings; A2. Obtain the loss function value corresponding to each hyperparameter encoding, and select the hyperparameter encoding with the smallest loss function as the optimal encoding; A3. Adopting adaptive search step size to perform adaptive multi-coding guided optimization on hyperparameter coding to increase population diversity and obtain hyperparameter coding after initial optimization; A4, using an adaptive search step size, and adaptively adjusting and optimizing the hyperparameter encoding after the initial optimization according to the optimal encoding, to obtain the hyperparameter encoding after the secondary optimization; A5. Perform local optimal development optimization on the hyperparameter encoding after the secondary optimization according to the optimal encoding to obtain the hyperparameter encoding after the tertiary optimization; A6. Re-obtain the optimal encoding based on the hyperparameter encoding after three optimizations; A7. Repeat steps A3 to A6 until the number of training times reaches the maximum number of training times, and use the current optimal encoding as the final hyperparameter of the machine vision model to obtain a pre-trained machine vision model.

8. The smart glasses system based on machine vision and voice interaction according to claim 6, characterized in that: Determine whether the predicted box of the object in the key frame of the video is within the restricted area, including: Determine the center point of the predicted box of the object in the video key frame to obtain the target center point; Determine the vertical center line and the horizontal center line of the video key frame; Determine whether the distance between the target center point and the vertical center line is less than a first distance threshold, if so, determine that the prediction box of the object in the video key frame is within the restricted area, otherwise further determine the relationship between the horizontal center line and the target center point; Determine whether the distance between the target center point and the vertical cross center line is less than the second distance threshold. If so, determine that the prediction box of the object in the video key frame is located in the restricted area; otherwise, determine that the prediction box of the object in the video key frame is not located in the restricted area.

9. The smart glasses system based on machine vision and voice interaction according to claim 1, characterized in that: Based on the sound data, a large language model is used to generate voice interaction data, including: Recognize the sound data collected by the sound collection module to obtain target text information; The target text information is used as input of a large language model to generate voice interaction data.

10. The smart glasses system based on machine vision and voice interaction according to claim 1, characterized in that: The large language analysis module is also used for: Recognize the sound data collected by the sound collection module to obtain target text information; The target text information is matched with the pre-stored control instructions by similarity, and when there is a control instruction with a similarity matching degree greater than a preset threshold, the control instruction with the greatest similarity is used as the target instruction; The target instruction is transmitted to the smart glasses so that the smart glasses perform corresponding operations.