Visually impaired person out-of-user travel system based on target detection technology and application thereof

Abstract

The invention relates to the technical field of visual impaired people equipment, in particular to a visual impaired people out-of-user travel system based on a target detection technology and application thereof. Comprising a core processing module and an intelligent bracelet module, the core processing module is used for detecting the front environment; the intelligent bracelet module is used for vibrating at the wrist of a user according to data detected by the core processing module, the core processing module comprises a shell, a cooling fan is fixed to the top of the shell, a first data processing board is fixed in the top end of the shell, and a start button is fixed in one side of the shell; and a second data processing board is fixed in the shell and located at the top of the first data processing board. According to the visual impaired person out-of-user travel system based on the target detection technology and the application thereof provided by the invention, the response speed of dangerous information feedback is improved through a method of combining local visual model identification and distance detection; in addition, through the combination of multiple functions, the safety of the visually impaired patient is guaranteed more comprehensively.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment for the visually impaired, and in particular to an outdoor travel system for the visually impaired based on target detection technology and its application. Background Art

[0002] In order to protect the daily safety of visually impaired patients, there are currently two main solutions in this field:

[0003] 1. Use ultrasonic ranging technology to convert the distance information in front into a vibration signal, allowing visually impaired patients to perceive nearby obstacles in real time.

[0004] 2. Use the camera to capture the image in front of you, use a large cloud model to describe the image content, and then convert it into voice to broadcast to the visually impaired patient.

[0005] The defects of the above technology are as follows:

[0006] Using ultrasonic distance measurement: Visually impaired patients will encounter various complex road conditions when going out. However, the ultrasonic distance measurement method is powerless to deal with potholes and bumps on the road, interruptions and loss of blind paths, etc. For example, in the actual use of ultrasonic bracelets for visually impaired patients, visually impaired patients need to actively place the device in a suitable position. The device uses ultrasonic distance measurement to convert the distance information into bracelet vibrations to achieve feedback on the distance in front. However, this method can only perform simple distance detection, while most of the time, a blind stick can better detect the distance, and the blind stick is more operable and provides clear feedback.

[0007] Method using camera + cloud-based large model: This method mostly uses the smart glasses for visually impaired patients, integrating the camera, speaker, and signal transmitter on the glasses. When in use, the camera captures the picture in front of the visually impaired patient and transmits it to the cloud-based large model. After the large model describes the picture, it is converted into voice and broadcast through the speaker. This method is highly dependent on the network, and the large model currently processes the text description of the video stream slowly, with a delay of more than 5 seconds. At the same time, most of the time the large model cannot correctly solve the problem. This instability makes it difficult to provide real help for visually impaired patients.

[0008] To this end, an outdoor travel system for visually impaired people based on target detection technology and its application are designed to provide another technical solution to the above technical problems. Summary of the invention

[0009] Based on this, it is necessary to provide an outdoor travel system and its application for visually impaired users based on target detection technology to solve the technical problems raised in the above background technology.

[0010] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0011] Outdoor travel system for visually impaired users based on target detection technology, including core processing module and smart bracelet module;

[0012] The core processing module is used to detect the environment ahead;

[0013] The smart bracelet module is used to vibrate the user's wrist according to the data detected by the core processing module.

[0014] As a preferred implementation of the outdoor travel system for visually impaired users based on target detection technology provided by the present invention, the core processing module includes a shell, a cooling fan is fixed to the top of the shell, a first data processing board is fixed inside the top of the shell, a start button is fixed inside one side of the shell, a second data processing board is fixed inside the shell and on the top of the first data processing board, one end of the first data processing board is connected to a GPS module via a data cable, and one end of the first data processing board is also connected to a depth sensing camera via a data cable.

[0015] As a preferred implementation of the outdoor travel system for visually impaired users based on target detection technology provided by the present invention, the material of the shell is acrylic.

[0016] As a preferred implementation of the outdoor travel system for visually impaired users based on target detection technology provided by the present invention, a lithium battery is fixed to the other end of the cooling fan, and the lithium battery and the first data processing board, the lithium battery and the second data processing board, the lithium battery and the GPS module, the lithium battery and the depth sensing camera, and the lithium battery and the cooling fan are all electrically connected through wires.

[0017] As a preferred implementation of the outdoor travel system for visually impaired users based on target detection technology provided by the present invention, the smart bracelet module includes a fixed shell, the top of the fixed shell is slidably connected to a cover plate, a side panel is fixed inside one end of the fixed shell, a PCB core board is fixed inside the fixed shell, a status indicator light is fixed on one side of one end of the top of the PCB core board, a communication chip is fixed on the other side of one end of the top of the PCB core board, the other end of the top of the PCB core board is fixed with a first flashing system button, a second flashing system button, a motion processing unit and a motor transmission interface in sequence, and patch vibration motors are evenly fixed inside the fixed shell and on the outside of the PCB core board.

[0018] As a preferred implementation of the outdoor travel system for visually impaired users based on target detection technology provided by the present invention, a battery is also fixed inside the fixed shell, and the battery and the patch vibration motor, the battery and the motion processing unit, the battery and the communication chip, and the battery and the PCB core board are all electrically connected through wires.

[0019] Application of an outdoor travel system for visually impaired persons based on target detection technology, for any of the above, applying the outdoor travel system for visually impaired persons in outdoor dangerous scenarios;

[0020] Apply the outdoor travel system for visually impaired people to the desktop object detection scenario.

[0021] As a preferred implementation of the application of the outdoor travel system for visually impaired persons based on the target detection technology provided by the present invention, the outdoor travel system for visually impaired persons is applied to outdoor dangerous scenes, and the steps are as follows:

[0022] After the user wears the core processing module and the smart bracelet module and turns them on, the lithium battery and storage battery are used to power each part;

[0023] The depth sensing camera will capture the image and depth data in front of the user in real time and transmit it to the first data processing board via a data line;

[0024] The first data processing board uses the target detection model to identify the input image as a possible dangerous target, and further constrains the recognition result through the depth data of the depth sensing camera;

[0025] If the first data processing board determines that there is a dangerous target at the current moment, it will convert the dangerous type, location information, and urgency of the dangerous target into motor vibration information and transmit it to the second data processing board;

[0026] The second data processing board transmits the motor vibration information to the communication chip of the smart bracelet module; the communication chip controls each patch vibration motor to vibrate with different frequencies and amplitudes through the motor transmission interface;

[0027] The user feels the vibration on the wrist and understands the type of danger, location information, and urgency of the dangerous target in front of him.

[0028] As a preferred implementation of the application of the outdoor travel system for visually impaired users based on the target detection technology provided by the present invention, the outdoor travel system for visually impaired users is applied to a desktop object detection scenario, and the steps are as follows:

[0029] After the user wears the core processing module and the smart bracelet module and turns them on, the lithium battery and storage battery are used to power each part;

[0030] The user removes the depth sensing camera from the core processing module and installs it on the smart bracelet module; the first data processing board will automatically enter the desktop object detection mode;

[0031] The depth sensing camera will capture the depth data in front of it in real time and transmit it to the first data processing board through the data line;

[0032] The first data processing board uses a desktop detection program to perform a convolution operation on the input depth data to find the object contour and obtain only the distance and position of the nearest object;

[0033] If the first data processing board detects an object on the desktop, it will transmit the distance and position of the nearest object on the desktop as motor vibration information to the second data processing board;

[0034] The second data processing board transmits the motor vibration information to the communication chip of the smart bracelet module; the communication chip controls each patch vibration motor to vibrate with different frequencies and amplitudes through the motor transmission interface;

[0035] Users can feel the motor vibration at their wrist to understand the distance and position of the nearest object on the desktop.

[0036] It can be seen without a doubt that the above-mentioned technical solution of the present application can definitely solve the technical problem to be solved by the present application.

[0037] At the same time, through the above technical solutions, the present invention has at least the following beneficial effects:

[0038] 1. The outdoor travel system for visually impaired users based on target detection technology and its application provided by the present invention improves the response speed to danger information feedback through a method combining local visual model recognition with distance detection; in addition, through the combination of multiple functions, the safety of visually impaired patients is more comprehensively guaranteed.

[0039] 2. The present invention has more efficient and flexible computing capabilities and a low-latency visual model on the terminal side, which can enable the wearer to receive low-latency notifications of dangerous road conditions and multi-information vibration notifications during travel, thereby ensuring travel safety and achieving good passability and safety on bumpy roads, zebra crossings, etc.

[0040] 3. The present invention senses the 3D environment through the camera on the worn backpack and transmits the RGB image to the computing unit in the backpack for processing, converts it into a directional vibration command, and transmits it to the bracelet through the network protocol to complete the notification of danger information and other information.

[0041] 4. The present invention uses the ESPNOW network communication protocol to bypass various complex transmission protocol layers. In complex scenarios with noisy channels, the computing unit can still transmit vibration instructions to the bracelet through the protocol to complete the notification of important information, greatly increasing the stability of the device.

[0042] 5. The computing unit of the present invention adopts the YoloV8 visual model in combination with the scene depth information obtained by 3D structured light, which enables the computing unit to quickly infer and reconstruct the scene information and accurately analyze important road conditions and dangerous information. The recognition rate increases exponentially during the travel process, providing safety protection for the wearer. At the same time, the computing unit and the bracelet are combined to provide the wearer with fast scene recognition and reconstruction, accurate information communication, and escort visually impaired patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 This is a schematic diagram of the structure modeling of the core module of the device of the present invention;

[0045] Figure 2 It is an exploded diagram of the core module structure of the device of the present invention;

[0046] Figure 3 This is a schematic diagram of a field wearing demonstration of the device of the present invention;

[0047] Figure 4 This is a modeling diagram of the smart bracelet module of the present invention;

[0048] Figure 5 This is an exploded view of the smart bracelet module of the present invention;

[0049] Figure 6 This is a schematic diagram of the PCB core structure inside the bracelet of the present invention;

[0050] Figure 7 This is a schematic diagram of a field wearing demonstration of the smart bracelet of the present invention;

[0051] Figure 8 This is a schematic diagram showing the output of the smart bracelet of the present invention;

[0052] Fig. 9 is a system architecture diagram of the present invention;

[0053] Fig.10 This is a schematic diagram showing an example of the outdoor dangerous scene detection function of the present invention;

[0054] Fig.11 It is a schematic diagram of data set quality comparison under the same model of the present invention;

[0055] Fig.12 A schematic diagram showing the comparison between the network architecture of the present invention before and after optimization;

[0056] Fig.13 Schematic diagram of mAP50-90 before and after optimization of yo l ov8 of the present invention;

[0057] Fig.14 It is a schematic diagram of F1 before and after yo l ov8 optimization of the present invention;

[0058] Fig.15 This is a schematic diagram of an example of a dangerous red light scene on a pedestrian crossing according to the present invention;

[0059] Fig.16 It is a schematic diagram showing the concentration of pheromones in the danger zone of the present invention;

[0060] Fig.17 This is a schematic diagram showing the desktop detection mode of the present invention;

[0061] Fig.18 This is a schematic diagram of speed optimization after the system of the present invention is worn.

[0062] In the figure: 100, outer shell; 110, cooling fan; 120, start button; 200, first data processing board; 210, second data processing board; 220, GPS module; 230, lithium battery; 240, depth sensing camera; 300, fixed shell; 310, side panel; 320, cover plate; 400, PCB core board; 410, first flashing system button; 420, second flashing system button; 430, power switch; 440, communication chip; 450, motion processing unit; 460, motor transmission interface; 470, status indicator light. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0065] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0066] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0067] Embodiment 1

[0068] Reference Figure 1-Figure 8 , an outdoor travel system for visually impaired users based on target detection technology, including a core processing module and a smart bracelet module;

[0069] The core processing module is used to detect the environment ahead, and its appearance can be as follows Figure 3 As shown, it is in the shape of a shoulder bag;

[0070] The smart bracelet module is used to vibrate the user's wrist according to the data detected by the core processing module. Its appearance can be as follows Figure 7 and Figure 8 As shown, it is in the form of a watch;

[0071] The core processing module includes a housing 100, which is made of acrylic, so that the housing 100 can stabilize and protect the device. A cooling fan 110 is fixed on the top of the housing 100, so that the cooling fan 110 can achieve a heat dissipation effect. A first data processing board 200 is fixed inside the top of the housing 100, so that the first data processing board 200 is used as a computing unit core for data processing. A start button 120 is fixed inside one side of the housing 100, so that the first data processing board 200 is started by the start button 120;

[0072] A second data processing board 210 is fixed inside the housing 100 and on the top of the first data processing board 200, so that the vibration signal can be transmitted to the smart bracelet module through the second data processing board 210. One end of the first data processing board 200 is connected to a GPS module 220 through a data line, and then the core processing module provides positioning through the GPS module 220. One end of the first data processing board 200 is also connected to a depth sensing camera 240 through a data line, so that the front picture and depth data can be captured as input. The other end of the cooling fan 110 A lithium battery 230 is fixed, and the lithium battery 230 and the first data processing board 200, the lithium battery 230 and the second data processing board 210, the lithium battery 230 and the GPS module 220, the lithium battery 230 and the depth sensing camera 240, and the lithium battery 230 and the cooling fan 110 are all electrically connected through wires, so that the lithium battery 230 can provide power to the cooling fan 110, the first data processing board 200, the second data processing board 210, the GPS module 220, and the depth sensing camera 240, and transmit data to each other;

[0073] In this embodiment, a strap is installed on the housing 100 so that the core processing module is formed into a small bag shape, which is carried on the body, and the depth sensing camera 240 is fixed to the strap through a back clip.

[0074] In this embodiment, the first data processing board 200 is a Jetson Nano development board, the second data processing board 210 is an ESP-8266 development board, and the model of the depth sensing camera 240 is Real sense D435.

[0075] The smart bracelet module includes a fixed shell 300, and a cover plate 320 is slidably connected to the top of the fixed shell 300, so that the cover plate 320 can be opened by sliding from one end of the top of the fixed shell 300, and the cover plate 320 and the fixed shell 300 are fixed by magnetic attraction. A side plate 310 is fixed inside one end of the fixed shell 300, and the inside of the fixed shell 300 can be viewed through the position of the side plate 310. A PCB core board 400 is fixed inside the fixed shell 300, and data sent by the core processing module can be received through the PCB core board 400, and various processing and control of the smart bracelet module can be realized;

[0076] A status indicator light 470 is fixed to one side of one end of the top of the PCB core board 400, so that the status indicator light 470 can provide information reminders when charging and the battery is low. A communication chip 440 is fixed to the other side of the top of the PCB core board 400, so that data can be transmitted to each other with the second data processing board 210 in the core processing module through the communication chip 440. The other end of the top of the PCB core board 400 is fixed with a first flashing system button 410, a second flashing system button 420, a motion processing unit 450 and a motor transmission interface 460 in sequence from one side to the other side, so that the PCB core board 400 is system-flashed through the first flashing system button 410 and the second flashing system button 420, and the user's motion state and overall acceleration are obtained through the motion processing unit 450 to predict and estimate the user's movements, etc. The motor transmission interface 460 provides power transmission for the vibration motor interface, and converts the received danger information into a vibration signal and transmits it to the user's wrist;

[0077] Patch vibration motors are evenly fixed inside the fixed shell 300 and on the outside of the PCB core board 400, which are used to vibrate through the control of the motor transmission interface 460. A battery is also fixed inside the fixed shell 300. The battery and the patch vibration motor, the battery and the motion processing unit 450, the battery and the communication chip 440, and the battery and the PCB core board 400 are all electrically connected through wires, and the battery provides power to the PCB core board 400, the communication chip 440, the motion processing unit 450 and the patch vibration motor.

[0078] In this embodiment, the communication chip 440 is an ESP8266 MCU communication chip.

[0079] Embodiment 2

[0080] Based on the above-mentioned embodiment 1, the outdoor travel system for visually impaired people is applied in outdoor dangerous scenes.

[0081] Working principle:

[0082] 1. After the user wears the core processing module and the smart bracelet module and turns them on, the lithium battery 230 and the storage battery are used to power each part;

[0083] 2. The depth sensing camera 240 will capture the image and depth data in front of the user in real time and transmit it to the first data processing board 200 via a data line;

[0084] 3. The first data processing board 200 uses the target detection model to identify the input image as a possible dangerous target, and then further constrains the recognition result through the depth data of the depth sensing camera 240.

[0085] 4. If the first data processing board 200 determines that there is a dangerous target at the current moment, it will convert the dangerous type, location information, and urgency of the dangerous target into motor vibration information and transmit it to the second data processing board 210.

[0086] 5. The second data processing board 210 transmits the motor vibration information to the communication chip 440 of the smart bracelet module. The communication chip 440 controls each patch vibration motor to vibrate with different frequencies and amplitudes through the motor transmission interface 460.

[0087] 6. The user feels the vibration on the wrist and understands the type of danger, location information, and urgency of the dangerous target in front of him.

[0088] Directions:

[0089] The user wears the core processing module and the smart bracelet module, presses the start button 120 of the core processing module and the power switch 430 of the smart bracelet module, and after waiting for about 20 seconds, the full motor of the smart bracelet module vibrates twice to indicate that the device has started to work normally;

[0090] When users go out and encounter dangerous situations such as interrupted blind paths, traffic lights, potholes on the road, missing sidewalks, and too close distances, they can feel the motor vibration of the smart bracelet module on their wrists, and promptly understand the type of danger, location information, and urgency of the dangerous target in front of them, and avoid the danger in advance;

[0091] After use, press the start button 120 and the power switch 430, and the smart bracelet module vibrates once to indicate that the device has been safely shut down; the user takes off the device.

[0092] Embodiment 3

[0093] Based on the above-mentioned embodiment 1, the outdoor travel system for visually impaired people is applied to the desktop object detection scenario.

[0094] Working principle:

[0095] 1. After the user wears the core processing module and the smart bracelet module and turns them on, the lithium battery 230 and the storage battery are used to power each part;

[0096] 2. The user removes the depth sensing camera 240 from the core processing module and installs it on the smart bracelet module; the first data processing board 200 will automatically enter the desktop object detection mode.

[0097] 3. The depth sensing camera 240 will capture the depth data in front of it in real time and transmit it to the first data processing board 200 via the data line;

[0098] 4. The first data processing board 200 uses a desktop detection program to perform a convolution operation on the input depth data to find the object contour and only obtain the distance and position of the nearest object;

[0099] 5. If the first data processing board 200 detects an object on the desktop, it will transmit the distance and position of the nearest object on the desktop as motor vibration information to the second data processing board 210;

[0100] 6. The second data processing board 210 transmits the motor vibration information to the communication chip 440 of the smart bracelet module; the communication chip 440 controls each patch vibration motor to vibrate with different frequencies and amplitudes through the motor transmission interface 460;

[0101] 7. Users can understand the distance and position of the nearest object on the desktop by feeling the motor vibration at their wrist.

[0102] Directions:

[0103] The user wears the device, presses the start button 120 of the core processing module and the power switch 430 of the smart bracelet module, removes the depth sensing camera 240 from the core processing module and installs it on the smart bracelet module. After waiting for about 20 seconds, the smart bracelet full motor vibrates twice to indicate that the device has started to work normally.

[0104] The user extends the hand wearing the smart bracelet module above the table, and senses the location of the table items by feeling the vibration of the motor in different directions on the wrist, and senses the distance of the table items from the hand by feeling the amplitude of the motor. The user follows the guidance of the smart bracelet to safely and accurately pick up the table items.

[0105] After use, press the start button 120 of the core processing module and the power switch 430 of the smart bracelet module, and the smart bracelet full motor vibrates once to indicate that the device has been safely shut down. The user takes off the device.

[0106] Embodiment 4

[0107] On the basis of the above-mentioned Embodiment 1, Embodiment 2 and Embodiment 3, other optimization schemes are disclosed.

[0108] 1. Optimize the model to achieve more efficient and accurate recognition.

[0109] 2. Design the software and bind it to the user’s mobile phone.

[0110] 3. The camera module is optimized for wireless transmission.

[0111] 4. The overall appearance design of the equipment is optimized, making it more concise and reliable.

[0112] 5. Change the bracelet vibration unit, from the original 4 single-frequency vibration units to 8 variable-frequency vibration elements.

[0113] 6. Add environmental monitoring function, collect information from the microphone of the bracelet and the backpack camera to identify the wearer's environment, so as to more accurately judge road conditions and danger information.

[0114] Embodiment 5

[0115] Based on the above embodiment 1, reference is made to Figure 9-Figure 18 ,Outdoor travel system for visually impaired people based on object detection technology.

[0116] 1. Structural function

[0117] The core function of this system is to perceive dangerous scenes in real time and issue timely warnings through depth sensing cameras 240, edge computing and vibration feedback.

[0118] The main architecture of the system is as follows Fig. 9 As shown, it mainly includes three functions: outdoor dangerous scene detection, memory and sharing of dangerous locations, and desktop object detection.

[0119] Outdoor dangerous scene detection: The system uses the target depth information of the depth sensing camera 240 to constrain the detection results of the target detection model, and transmits the signal to the smart bracelet module. The user can accurately know the dangerous scene through the vibration of the smart bracelet module in different directions and amplitudes.

[0120] Memory and sharing of dangerous locations: After the user confirms the dangerous location, the system can store and share the current dangerous location, and the bracelet will prompt when passing through this area again. At the same time, when navigating, the route will recommend areas with lower danger concentration.

[0121] Desktop object detection: The system uses the depth information of the depth sensing camera 240 and realizes the distance and direction detection of desktop objects through morphological processing.

[0122] 2. Outdoor Dangerous Scene Detection

[0123] The device is equipped with a depth sensing camera 240, which has infrared function and can capture RGB images and deep infrared images at the same time, so as to identify obstacles and road signs such as traffic lights, zebra crossings, blind paths and other key information. With the help of the YOLOv8 target detection model, the device processes images in real time and identifies important information in front of the visually impaired patient. The model is deployed on the first data processing board 200 of the Jes i on Or in Nano development board, without relying on the cloud, ensuring that the device maintains real-time response in complex outdoor environments. Fig.10 shown.

[0124] 2.1 Dataset Creation

[0125] like Fig.11 As shown in the figure, in order to obtain a high-quality dataset that meets the goal, more than 3,000 photos were taken in multiple locations with handheld cameras, and data cleaning and annotation were performed to build a special dataset for blind people walking outdoors, so as to better identify road surfaces that affect travel, such as blind paths, traffic lights, zebra crossings, and potholes. Relying on high-quality datasets, the accuracy of the model has been greatly improved, which is better than other datasets on the market.

[0126] 2.2 Model Training

[0127] The basic model is YOLOv8. In this dataset, adding the EMA module to the original network can achieve better results.

[0128] EMA is a multi-scale parallel sub-network used to establish short- and long-term dependencies. An EMA module is added to the tail of the backbone, and an EMA module is added to each of the three locations of the detection head. The network architecture before and after optimization is shown in the figure below. Fig.12 ;

[0129] The optimized model was tested for 80 iterations, and the recognition accuracy was improved while the model parameters remained basically unchanged. Fig.13 This is the mAP50-90 comparison chart and the F1 curve comparison chart after 80 iterations of yo l ov8n:

[0130] At the same time, the outer layer of the model is improved to match the depth data captured by the depth sensing camera 240 with the identified target so that the distance of the target can be displayed in real time. The specific steps are as follows:

[0131] 1. Use the pyreal sense2 library to convert the depth frames captured by the depth sensing camera 240 into numpy arrays;

[0132] 2. For each target detected by yo l ov8 through the RGB frame, calculate the coordinates of the center point of its bounding box;

[0133] 3. Randomly sample multiple points around the center point and obtain the depth values ​​of these points;

[0134] 4. Perform median filtering on the sampled depth values ​​to obtain a more stable target depth estimation;

[0135] 5. Add the target depth value obtained in step 4 to the target frame;

[0136] 6. Disassemble and analyze various dangerous scenes, and use the target distance to constrain the original detection results. Take the partial constraints of the red light as an example, Fig.15 shown.

[0137] 2.3 Vibration feedback mechanism

[0138] In order to overcome the limitations of voice feedback in noisy environments, the device uses a vibration feedback mechanism. The bracelet is equipped with 8 AAC linear motors or patch vibration motors, which transmit road condition information to visually impaired patients through vibrations of different directions and intensities. This feedback method realizes efficient and instant information transmission and indicates directional changes in the surrounding environment through different vibration modes.

[0139] 2.4 Intelligent Communication and Data Transmission

[0140] The device uses the ESP-NOW communication protocol for data transmission, ensuring that vibration feedback information is transmitted to the smart bracelet with low latency and low power consumption. Compared with traditional Wi-Fi, ESP-NOW has stronger anti-interference ability and lower packet loss rate, and can maintain the stability of information transmission even at long distances or under signal interference.

[0141] 3. Memory and Sharing of Dangerous Locations

[0142] 3.1 Dangerous area marking and warning

[0143] When the device detects that the user has entered a dangerous area such as a pothole, obstacle, etc., the user can mark the location through the GPS module and store it in the local database. When the user approaches the marked dangerous area again, the device will automatically trigger an early warning to remind the user to ensure travel safety.

[0144] 3.2 Risk Map of Swarm Mechanism

[0145] After the device is connected to the network, the dangerous locations recorded locally are automatically uploaded to the server. The server uses the pheromone concentration in the bee colony algorithm to indicate the degree of danger in different areas, such as Fig.16 During navigation, the system selects a relatively safe path based on the pheromone concentration to help users avoid high-risk areas.

[0146] 4. Desktop object detection

[0147] To simplify user operations, the desktop mode is automatically switched: the system enters the desktop mode after recognizing a flat desktop for a long time. To reduce the amount of calculation and improve the response speed, only the depth information of the camera is called in the desktop mode. After the system finds a flat area, it performs a convolution operation on the flat area to find the outline of the object, and only obtains the distance and position of the nearest object, thereby guiding the visually impaired to pick up items safely.

[0148] After entering the desktop object detection mode, the wristband will continue to vibrate, and the depth sensing camera 240 will move from the backpack to the wristband position to start detecting desktop objects. When the depth sensing camera 240 recognizes an object, it will guide the user to detect it through different vibration intensities and directions. If the desktop is not detected for a long time, the system will automatically switch back to outdoor mode. Fig.17 So, show and Fig.17 The middle left, top and bottom are respectively: third-person real shot, camera image, and processed image.

[0149] 5. Effect

[0150] like Fig.18 As shown in the figure, the experiment sets up a visually impaired patient to calculate the time from point A to point B in the state of no equipment and wearing assistive devices, and records the state of wearing assistive devices and collects and adjusts the algorithm of visual model, so as to help the visually impaired patients in their daily travel.

[0151] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the present invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A system for outdoor travel for visually impaired users based on target detection technology, characterized in that: Including core processing module and smart bracelet module; The core processing module is used to detect the environment ahead; The smart bracelet module is used to vibrate the user's wrist according to the data detected by the core processing module.

2. The outdoor travel system for visually impaired users based on target detection technology according to claim 1, characterized in that: The core processing module comprises a housing (100), a cooling fan (110) is fixed on the top of the housing (100), a first data processing board (200) is fixed inside the top of the housing (100), a start button (120) is fixed inside one side of the housing (100), a second data processing board (210) is fixed inside the housing (100) and located on the top of the first data processing board (200), one end of the first data processing board (200) is connected to a GPS module (220) via a data cable, and one end of the first data processing board (200) is also connected to a depth sensing camera (240) via a data cable.

3. The outdoor travel system for visually impaired users based on target detection technology according to claim 2, characterized in that: The shell (100) is made of acrylic.

4. The outdoor travel system for visually impaired users based on target detection technology according to claim 2, characterized in that: A lithium battery (230) is fixed to the other end of the cooling fan (110); the lithium battery (230) and the first data processing board (200), the lithium battery (230) and the second data processing board (210), the lithium battery (230) and the GPS module (220), the lithium battery (230) and the depth sensing camera (240), and the lithium battery (230) and the cooling fan (110) are all electrically connected via wires.

5. The outdoor travel system for visually impaired users based on target detection technology according to claim 1, characterized in that: The smart bracelet module comprises a fixed shell (300), the top of the fixed shell (300) is slidably connected to a cover plate (320), a side plate (310) is fixed inside one end of the fixed shell (300), a PCB core board (400) is fixed inside the fixed shell (300), a status indicator light (470) is fixed on one side of the top end of the PCB core board (400), a communication chip (440) is fixed on the other side of the top end of the PCB core board (400), a first flashing system button (410), a second flashing system button (420), a motion processing unit (450) and a motor transmission interface (460) are fixed in sequence from one side to the other side of the top end of the PCB core board (400), and patch vibration motors are evenly fixed inside the fixed shell (300) and on the outside of the PCB core board (400).

6. The outdoor travel system for visually impaired users based on target detection technology according to claim 5, characterized in that: A storage battery is also fixed inside the fixed shell (300), and the storage battery and the patch vibration motor, the storage battery and the motion processing unit (450), the storage battery and the communication chip (440), and the storage battery and the PCB core board (400) are all electrically connected via wires.

7. Application of a system for visually impaired users to travel outdoors based on target detection technology, according to any one of claims 1 to 6, characterized in that: Apply the outdoor travel system for the visually impaired to outdoor dangerous scenarios; Apply the outdoor travel system for visually impaired people to the desktop object detection scenario.

8. The application of the outdoor travel system for visually impaired users based on target detection technology according to claim 7, characterized in that: To apply the outdoor travel system for the visually impaired in outdoor dangerous scenarios, the steps are as follows: After the user wears the core processing module and the smart bracelet module and turns them on, the lithium battery (230) and the storage battery are used to supply power to each part; The depth sensing camera (240) will capture the image and depth data in front of the user in real time, and transmit it to the first data processing board (200) via a data line; The first data processing board (200) uses a target detection model to identify a possible dangerous target in an input image, and further constrains the recognition result through depth data of a depth sensing camera (240); If the first data processing board (200) determines that a dangerous target exists at the current moment, the dangerous type, location information, and urgency of the dangerous target will be converted into motor vibration information and transmitted to the second data processing board (210); The second data processing board (210) transmits the motor vibration information to the communication chip (440) of the smart bracelet module; the communication chip (440) controls each patch vibration motor to vibrate with different frequencies and amplitudes through the motor transmission interface (460); The user feels the vibration on the wrist and understands the type of danger, location information, and urgency of the dangerous target in front of him.

9. The application of the outdoor travel system for visually impaired users based on target detection technology according to claim 7, characterized in that: The steps to apply the outdoor travel system for visually impaired people to the desktop object detection scenario are as follows: After the user wears the core processing module and the smart bracelet module and turns them on, the lithium battery (230) and the storage battery are used to supply power to each part; The user removes the depth sensing camera (240) from the core processing module and installs it on the smart bracelet module; the first data processing board (200) automatically enters the desktop object detection mode; The depth sensing camera (240) will capture the depth data in front of it in real time and transmit it to the first data processing board (200) via a data line; The first data processing board (200) uses a desktop detection program to perform a convolution operation on the input depth data to find the contour of the object and only obtain the distance and position of the nearest object; If the first data processing board (200) detects an object on the desktop, the distance and position of the nearest object on the desktop will be transmitted as motor vibration information to the second data processing board (210); The second data processing board (210) transmits the motor vibration information to the communication chip (440) of the smart bracelet module; the communication chip (440) controls each patch vibration motor to vibrate with different frequencies and amplitudes through the motor transmission interface (460); Users can feel the motor vibration at their wrist to understand the distance and position of the nearest object on the desktop.

Images