Dual-mode blind guiding auxiliary method and system

Through dual-mode blinding assistance methods and systems, the traffic marker recognition model is used to identify environmental images, calculate crossing distances or blind path obstacle distances, which solves the problem of visually impaired navigation in traffic environments, and improves environmental perception and navigation capabilities , achieving safer travel.

CN118918704APending Publication Date: 2024-11-08GUANGDONG OCEAN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410994689.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Visually impaired people find it difficult to identify the location and traffic light status of the road when crossing the road and walking. Traditional blinding methods such as guide dogs and blind guide sticks have limited navigation capabilities in complex traffic environments and cannot provide comprehensive environmental perception and navigation support, which affects Walk safely.

Method used

Dual-mode guide assistance methods and systems are adopted to obtain environmental images, use traffic sign recognition models to perform pattern recognition, calculate crossing distances or blind path obstacle distances, and provide blind path assistance through voice modules to provide safe crossing modes and safe blind paths. Mode navigation support.

Benefits of technology

It improves the environmental perception and navigation capabilities of visually impaired people, provides more comprehensive traffic environment perception and navigation support, and adapts to the needs of safe travel in complex traffic environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918704A_ABST
    Figure CN118918704A_ABST
Patent Text Reader

Abstract

The invention discloses a dual-mode blind guiding auxiliary method and system. The method comprises the following steps: acquiring an environment image; performing pattern recognition processing through the traffic marker recognition model to obtain a pattern recognition result; when the mode identification result is a safe crossing mode, extracting a target crossing image from the environment image; according to the target street-crossing image, calculating a street-crossing distance; according to the street crossing distance, blind guiding assistance is performed through a voice module; or; when the mode recognition result is a safe blind sidewalk mode, extracting a blind sidewalk image to be processed from the environment image; according to the to-be-processed blind sidewalk image, calculating a blind sidewalk obstacle distance; according to the blind sidewalk obstacle distance, blind guiding assistance is carried out through the voice module. According to the invention, environment perception is carried out through image processing, dual-mode blind guiding assistance is realized by identifying a safe street crossing mode or a safe blind sidewalk mode, and the environment perception capability and the navigation capability are improved. The method can be applied to the technical field of visual identification and blind guiding assistance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of visual recognition and blind-guiding assistance technology, and in particular to a dual-mode blind-guiding assistance method and system. Background Art

[0002] Due to the lack of visual perception, visually impaired people often find it difficult to identify the location of the road and the status of traffic lights when crossing the road. When walking on the blind path, they often encounter problems such as illegal occupation or poor maintenance of the blind path, which seriously affects the walking safety of the visually impaired. Traditional guidance methods use guide dogs or guide sticks for guidance, but these methods have obvious limitations. The training and maintenance costs of guide dogs are high, and there is a problem that visually impaired people find it difficult to adapt to traveling with guide dogs. In complex traffic environments, the navigation ability of guide sticks is limited, and it is difficult to provide comprehensive environmental perception and navigation support. The environmental perception ability is low and the navigation ability is low.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The embodiments of the present invention provide a dual-mode blind guide assistance method and system, which effectively improve the environmental perception capability and navigation capability.

[0005] On the one hand, an embodiment of the present invention provides a dual-mode blind guide assistance method, comprising the following steps:

[0006] Get the environment image;

[0007] According to the environmental image, pattern recognition processing is performed through a traffic sign recognition model to obtain a pattern recognition result, wherein the pattern recognition result includes a safe street crossing mode or a safe blind road mode;

[0008] When the pattern recognition result is the safe street crossing mode, extracting a target street crossing image from the environment image, wherein the target street crossing image includes a zebra crossing image or a traffic light image;

[0009] Calculating the street crossing distance according to the target street crossing image;

[0010] According to the crossing distance, the blind guide is assisted by a voice module;

[0011] or;

[0012] When the pattern recognition result is the safe blind path pattern, extracting a to-be-processed blind path image from the environment image;

[0013] Calculating the distance of obstacles on the blind path according to the blind path image to be processed;

[0014] According to the distance of the obstacle on the blind path, guidance assistance is provided through a voice module.

[0015] In some embodiments, the traffic sign recognition model is obtained by the following steps:

[0016] Get a traffic image set;

[0017] According to preset image requirements, the traffic image set is screened and processed;

[0018] Performing training set division processing on the traffic image set after screening processing to obtain a training set to be processed;

[0019] Performing noise reduction processing on the training set to be processed to obtain a first training set;

[0020] Performing contrast optimization processing on the first training set to obtain a second training set;

[0021] performing scale normalization processing on the second training set to obtain a third training set;

[0022] Performing random rotation processing on the third training set to obtain a fourth training set;

[0023] Performing random fuzzy processing on the fourth training set to obtain a fifth training set;

[0024] Performing cropping and adjustment processing on the fifth training set to obtain a target training set;

[0025] The target training set is input into a preset deep learning model so that the preset deep learning model is trained to obtain the traffic sign recognition model.

[0026] In some embodiments, performing pattern recognition processing based on the environment image through a traffic sign recognition model to obtain a pattern recognition result includes:

[0027] Inputting the environment image into the traffic sign recognition model for recognition to obtain a traffic sign recognition result;

[0028] When the traffic sign recognition result is a street crossing sign, taking the safe street crossing mode as the pattern recognition result;

[0029] When the traffic sign recognition result is a blind path sign, the safe blind path pattern is used as the pattern recognition result.

[0030] In some embodiments, the calculating the street crossing distance according to the target street crossing image includes:

[0031] The monocular camera is calibrated by the four-point calibration method to obtain the camera parameter matrix;

[0032] Constructing a ranging model according to the camera parameter matrix;

[0033] Extracting a frame image and identifying a target frame from the target street crossing image;

[0034] According to the recognition target frame, identifying the zebra crossing pixel width of the zebra crossing marker from the frame image;

[0035] Calculating the focal length according to the distance measurement model, the object reference width, the reference distance and the zebra crossing pixel width;

[0036] The street crossing distance is calculated according to the focal length, the object measurement width and the zebra crossing pixel width.

[0037] In some embodiments, providing blind-guiding assistance through a voice module according to the street crossing distance includes:

[0038] When the target street crossing image is the zebra crossing image, determining whether the street crossing distance is greater than a preset road edge distance;

[0039] If the crossing distance is greater than the preset road edge distance, the voice information of not crossing the street is broadcasted through the voice module;

[0040] If the crossing distance is not greater than the preset road edge distance, the voice information of crossing the street is broadcasted through the voice module;

[0041] When the target street crossing image is the traffic light image, the upcoming crossing voice information is announced through the voice module according to the street crossing distance.

[0042] In some embodiments, the step of calculating the distance of obstacles in the blind path according to the blind path image to be processed includes:

[0043] Performing morphological processing on the blind path image to be processed to obtain a morphological image;

[0044] Performing color feature extraction processing on the morphological image to obtain a target blind path image;

[0045] Extracting a blind path area from the target blind path image, wherein the blind path area includes a normal blind path area or an abnormal blind path area;

[0046] Calculate the obstacle measurement width according to the blind path measurement width, the obstacle pixel width and the blind path pixel width of the blind path area;

[0047] The obstacle distance of the blind path is calculated according to the measured width of the obstacle.

[0048] In some embodiments, the providing blind-guiding assistance through a voice module according to the distance of the blind path obstacle includes:

[0049] When the blind path area is the normal blind path area, the normal blind path voice information is broadcasted through the voice module;

[0050] When the blind path area is the blind path abnormal area, the blind path abnormal voice information is broadcasted through the voice module according to the blind path obstacle distance.

[0051] On the other hand, an embodiment of the present invention provides a dual-mode blind guide assistance system, including:

[0052] An image acquisition module, wherein the image acquisition module is used to acquire an environment image;

[0053] A scene judgment module, the scene judgment module is used to perform pattern recognition processing according to the environmental image through a traffic sign recognition model to obtain a pattern recognition result, wherein the pattern recognition result includes a safe street crossing mode or a safe blind road mode;

[0054] A safe street crossing module, the safe street crossing module is used to calculate the street crossing distance according to the safe street crossing mode;

[0055] A safety blind path module, the safety blind path module is used to calculate the distance of the blind path obstacle according to the safety blind path mode;

[0056] A voice module is used to broadcast the position voice information according to the target distance information to achieve blind guide assistance, and the target distance information includes the street crossing distance or the blind path obstacle distance.

[0057] In some embodiments, the safe crossing module includes:

[0058] A traffic sign recognition submodule, the traffic sign recognition submodule is used to extract a target street crossing image from the environment image, the target street crossing image includes a zebra crossing image or a traffic light image;

[0059] A monocular distance measurement submodule, the monocular distance measurement submodule is used to calculate the distance of the identified object according to the target crossing street image;

[0060] The position information processing submodule is used to calculate the street crossing distance according to the distance of the identification object.

[0061] In some embodiments, the safety blind path module includes:

[0062] An image processing submodule, the image processing submodule is used to extract the to-be-processed blind path image from the environment image, and perform preset image processing to obtain a target blind path image, wherein the preset image processing includes color feature extraction processing or morphological processing;

[0063] A blind path extraction submodule, the blind path extraction submodule is used to extract a blind path area from the target blind path image, the blind path area includes a normal blind path area or an abnormal blind path area;

[0064] The abnormal position detection submodule is used to calculate the distance of the obstacle of the blind path according to the blind path area.

[0065] The beneficial effects of the present invention are as follows:

[0066] The present invention first acquires an environmental image, performs pattern recognition processing through a traffic sign recognition model to obtain a pattern recognition result, and when the pattern recognition result is a safe street crossing mode, extracts a target street crossing image from the environmental image, calculates the street crossing distance, and then performs blind guide assistance through a voice module; or, when the pattern recognition result is a safe blind path mode, extracts a blind path image to be processed from the environmental image, calculates the distance of obstacles on the blind path, and then performs blind guide assistance through a voice module, so that environmental perception can be performed through image processing, and dual-mode blind guide assistance can be achieved by identifying the safe street crossing mode or the safe blind path mode, thereby improving environmental perception capability and navigation capability.

[0067] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 This is a flow chart of a dual-mode blind-guiding assistance method according to an embodiment of the present invention;

[0070] Figure 2 This is a schematic diagram of a result of traffic sign recognition data according to an embodiment of the present invention;

[0071] Figure 3 A schematic diagram of a traffic light image according to an embodiment of the present invention;

[0072] Figure 4 A schematic diagram of a result of identifying distance information of a traffic sign according to an embodiment of the present invention;

[0073] Figure 5 A schematic diagram of an overall process of calculating the street crossing distance according to an embodiment of the present invention;

[0074] Figure 6 A schematic diagram of a blind path image to be processed according to an embodiment of the present invention;

[0075] Figure 7 A schematic diagram of a result of identifying obstacles on a blind path according to an embodiment of the present invention;

[0076] Figure 8 A schematic diagram of an overall process of calculating the distance of an obstacle on a blind path according to an embodiment of the present invention;

[0077] Fig. 9 A schematic diagram of an overall process of blind-guiding assistance according to an embodiment of the present invention;

[0078] Fig.10 The present invention is a schematic structural diagram of a dual-mode blind guide assistance system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0080] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0081] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0082] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0083] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0084] K210 platform: refers to the Yabo Intelligent K210 development board. The software is based on Maxipy IDE, programmed in Python, and integrated with an artificial intelligence processing unit (AI Accelerator). It has powerful computing and reasoning capabilities, making the K210 platform have great potential in assisting the blind to identify the environment ahead and detect obstacles.

[0085] Microcontroller: STM32F103C8T6 is selected. The software is based on Keil 5 and programmed in C language. It has high performance, low power consumption, rich peripheral support, good stability and cost performance. It is equipped with a Cortex-M3 core and runs at a frequency of up to 72MHz, which enables it to efficiently process the control logic in the guide assistance system and coordinate the work of various modules.

[0086] Audio device: Select MP3-TF-16P, which provides a serial port interface. You can control the module to play the specified music by sending simple serial port commands, and realize various playback functions, such as play, pause, stop, etc. It provides a friendly voice prompt function for the guide assistance system to help the blind better understand the surrounding environment.

[0087] In the related art, due to the lack of visual perception, the visually impaired often find it difficult to judge the position of the road and identify the status of traffic lights when crossing the road. At the same time, when walking on the blind path, they often encounter problems such as illegal occupation or poor maintenance of the blind path, which seriously affect their walking safety. Traditional guide means, such as guide dogs and guide sticks, provide help to the visually impaired to a certain extent. However, these means also have obvious limitations. The training and maintenance costs of guide dogs are high, and not all visually impaired people can adapt to traveling with guide dogs. Although the guide stick is simple and easy to use, its navigation ability is limited in a complex traffic environment, and it cannot provide comprehensive environmental perception and navigation support. It cannot meet the needs of safe travel in a modern traffic environment, and has low environmental perception and navigation capabilities. Therefore, it is particularly important to develop a guide assistance system that can assist the visually impaired to travel safely.

[0088] In view of this, the present invention provides assistance for the safe travel of the blind. The traffic sign recognition model is trained based on the YOLOv2 algorithm, and the traffic sign recognition model is used to identify the main traffic signs such as traffic lights, zebra crossings, and blind paths. The distance information of the road distance, obstacles on the road ahead of the blind, and abnormal areas of the blind path are detected on the K210 platform to realize the distance perception and obstacle avoidance functions. The STM32 microcontroller receives the information of the K210 platform and sends it to the voice module, and tells the blind the information of traffic signs and obstacle distance information in real time in the form of voice, thereby improving the environmental perception and navigation capabilities. At the same time, an innovative dual-mode guide assistance system is designed, namely the safe crossing mode and the safe blind path mode, which is more suitable for the main travel scenarios of the blind.

[0089] A dual-mode blind guide assistance method provided in an embodiment of the present application relates to the field of visual recognition and blind guide assistance technology. A dual-mode blind guide assistance method provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application that implements a dual-mode blind guide assistance method, etc., but is not limited to the above forms.

[0090] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0091] The following is a detailed explanation of the embodiments of the present application in conjunction with the accompanying drawings:

[0092] Figure 1 is an optional flowchart of a dual-mode blind guide assistance method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S110.

[0093] Step S101, obtaining an environment image;

[0094] Step S102: performing pattern recognition processing through a traffic sign recognition model according to the environment image to obtain a pattern recognition result, wherein the pattern recognition result includes a safe street crossing mode or a safe blind road mode;

[0095] Step S103: When the mode recognition result is a safe street crossing mode, execute steps S104 to S106;

[0096] Step S104, extracting a target street crossing image from the environment image, where the target street crossing image includes a zebra crossing image or a traffic light image;

[0097] Step S105, calculating the street crossing distance according to the target street crossing image;

[0098] Step S106: providing blind-guiding assistance through a voice module according to the distance across the street;

[0099] Step S107: When the pattern recognition result is a safe blind road pattern, execute steps S108 to S110;

[0100] Step S108, extracting the blind path image to be processed from the environment image;

[0101] Step S109, calculating the distance of obstacles on the blind path according to the image of the blind path to be processed;

[0102] Step S110: providing blind-guiding assistance through a voice module according to the distance of obstacles on the blind path.

[0103] Steps S101 to S110 shown in the embodiment of the present application implement blind guide assistance and improve environmental perception and navigation capabilities.

[0104] In step S101 of some embodiments, the environment image may be acquired through a camera device of the K210 platform, or may be acquired through other methods, not limited thereto.

[0105] In some embodiments, in step S102, pattern recognition processing is performed based on the environment image through the traffic sign recognition model to obtain a pattern recognition result, which may include but is not limited to the following steps:

[0106] Inputting the environment image into the traffic sign recognition model for recognition, and obtaining the traffic sign recognition result;

[0107] When the traffic sign recognition result is a street crossing sign, the safe street crossing mode is used as the pattern recognition result;

[0108] When the traffic sign recognition result is a blind path sign, the safe blind path mode is used as the pattern recognition result.

[0109] In some embodiments, pattern recognition processing can be performed on the environment image through a traffic sign recognition model to obtain a pattern recognition result. The pattern recognition result includes a safe street crossing mode or a safe blind road mode. The environment image can be first input into the trained traffic sign recognition model for recognition, and the traffic signs can be recognized with high accuracy to obtain a traffic sign recognition result. The traffic sign recognition result may include a street crossing sign or a blind road sign. When the traffic sign recognition result is a street crossing sign (such as a zebra crossing, such as a traffic light), it indicates a road crossing scene. It can be determined that the blind person is currently in a street crossing state, and the safe street crossing mode is used as the pattern recognition result. When the traffic sign recognition result is a blind road sign, it can be determined that the blind person is currently in a blind road walking state, and the safe blind road mode is used as the pattern recognition result to facilitate the subsequent calculation of the two different modes. In addition, the traffic sign recognition result is as follows: Figure 2 As shown, the recognition accuracy of traffic lights is 0.99, and the recognition accuracy of sidewalks is 0.91.

[0110] In some embodiments, in step S102, obtaining a traffic sign recognition model may include but is not limited to the following steps:

[0111] Get a traffic image set;

[0112] According to the preset image requirements, the traffic image set is screened and processed;

[0113] Performing training set division processing on the filtered traffic image set to obtain a training set to be processed;

[0114] Perform noise reduction processing on the training set to be processed to obtain a first training set;

[0115] Performing contrast optimization processing on the first training set to obtain a second training set;

[0116] Performing scale normalization on the second training set to obtain a third training set;

[0117] Performing random rotation processing on the third training set to obtain a fourth training set;

[0118] Perform random fuzzy processing on the fourth training set to obtain the fifth training set;

[0119] The fifth training set is trimmed and adjusted to obtain a target training set;

[0120] The target training set is input into a preset deep learning model so that the preset deep learning model is trained to obtain a traffic sign recognition model.

[0121] In some embodiments, a traffic image set may be first obtained, wherein the traffic image set is composed of key traffic features in the daily life of the blind, and may include traffic lights, zebra crossings, blind paths, and road obstacles, one part of which is taken on-site, and one part is taken from a network high-definition image. Then, according to the preset image requirements, the traffic image set is screened and processed to remove image data that does not meet the requirements. Then, the filtered traffic image set is divided into a training set, and the training set to be processed may be divided according to the ratio of training set: verification set = 7:3 to obtain a training set to be processed. Then, the divided training set to be processed and the verification set to be processed are processed, and the training set to be processed is subjected to noise reduction processing to obtain a first training set, and the first training set is subjected to contrast optimization processing to obtain a second training set, and the second training set is subjected to scale normalization processing to obtain a third training set, thereby improving the image quality and making the key features more prominent. Then, the third training set is subjected to random rotation processing to obtain a fourth training set, and the fourth training set is subjected to random blur processing to obtain a fifth training set, and the fifth training set is subjected to cropping and adjustment processing to obtain a target training set, so that the training data set can be expanded to improve the generalization ability of the model. Finally, the preset deep learning model can be set as the YOLOv2 algorithm model, and the target training set can be input into the preset deep learning model so that the preset deep learning model can be trained to obtain a traffic sign recognition model. Moreover, the obtained traffic sign recognition model can be mounted on the recognition module of the K210 platform for subsequent use. It can be understood that YOLOv2 is more accurate in prediction, faster, and can recognize more objects than YOLOv1. And the proposed training method-joint training algorithm can mix the two data sets together. Use a hierarchical view to classify objects, and use a huge amount of classification data set data to expand the detection data set, thereby mixing two different data sets.

[0122] In some embodiments, in step S103, the pattern recognition result may be judged, and when the pattern recognition result is a safe street crossing mode, steps S104 to S106 are executed.

[0123] In some embodiments, in step S104, a target crossing image may be extracted from the environment image. The target crossing image may include a zebra crossing image or a traffic light image. The extracted traffic light image may be Figure 3As shown, the recognition accuracy is 0.93 and the street crossing distance is 14. The extraction method may include but is not limited to image cropping, image enlargement or image reduction.

[0124] In some embodiments, in step S105, calculating the street crossing distance according to the target street crossing image may include but is not limited to the following steps:

[0125] The monocular camera is calibrated by the four-point calibration method to obtain the camera parameter matrix;

[0126] Construct a distance measurement model based on the camera parameter matrix;

[0127] Extract frame images and identify target frames from target street crossing images;

[0128] According to the recognition target frame, the pixel width of the zebra crossing of the zebra crossing marker is recognized from the frame image;

[0129] Calculate the focal length according to the ranging model, the object reference width, the reference distance and the pixel width of the zebra crossing;

[0130] Calculate the distance to cross the street based on the focal length, the measured width of the object, and the pixel width of the zebra crossing.

[0131] In some embodiments, the monocular camera can be calibrated by the four-point calibration method to obtain a camera parameter matrix. For example, a checkerboard can be prepared as a calibration object, and the calibration object can be photographed using the camera device of the K210 platform to ensure that the calibration object is clearly visible in the image and covers different positions and angles of the camera field of view. Using the four-point calibration method, the internal and external parameters of the camera are calculated by obtaining the lateral distances of the four calibration points to the camera and the simultaneous equations of their pixel coordinates to obtain the camera parameter matrix. Then, based on the camera parameter matrix, the pinhole imaging principle is used to construct a ranging model based on the internal and external parameters of the camera. Let the focal length be F, the object reference width of the object to be identified when training the traffic sign recognition model be W (cm), the zebra crossing pixel width of the training image of the object to be identified be p, and the reference distance between the object to be identified and the camera when training the traffic sign recognition model be D (cm). Then, on the basis of identifying the zebra crossing from the target crossing image, the frame image and the recognition target frame are extracted, and the zebra crossing pixel width of the zebra crossing marker is identified from the frame image according to the recognition target frame. For example, a rectangle search (single zebra crossing) with a specified threshold can be performed in the recognition target frame, and the width of the single zebra crossing with the lowest coordinates in the image frame is set as the zebra crossing pixel width P to improve the ranging accuracy. Then, the focal length is calculated based on the ranging model, the object reference width, the reference distance, and the zebra crossing pixel width. Among them, the object reference width and the reference distance are known values, and the zebra crossing pixel width can be obtained from the Pos array in the recognition module. The calculation formula for the focal length is: Where F is the focal length, P is the pixel width of the zebra crossing, D is the reference distance, and W is the reference width of the object. Finally, the crossing distance is calculated based on the focal length, the measured width of the object, and the pixel width of the zebra crossing. The calculation formula for the crossing distance is: Where D' is the distance across the street, and W' is the measured width of the object. It is understandable that after the focal length is known, for objects of standard size such as traffic signs, the known size can be used as the measured width W' of the object. For example, the distance information is as follows: Figure 4 As shown, when the recognized object is a traffic light, the calculated crossing distance is 6m and the recognition accuracy is 0.99; when the recognized object is a sidewalk, the calculated crossing distance is 3m and the recognition accuracy is 0.91.

[0132] In some embodiments, in step S106, according to the distance across the street, the blind guide assistance is provided by the voice module, which may include but is not limited to the following steps:

[0133] When the target street crossing image is a zebra crossing image, determining whether the street crossing distance is greater than a preset road edge distance;

[0134] If the crossing distance is greater than the preset road edge distance, the voice information of not crossing the street will be broadcasted through the voice module;

[0135] If the crossing distance is not greater than the preset road edge distance, the voice information of crossing the street will be broadcast through the voice module;

[0136] When the target street crossing image is a traffic light image, the upcoming voice information will be announced through the voice module according to the street crossing distance.

[0137] In some embodiments, the key traffic sign information identified by the traffic sign recognition model, such as zebra crossings and traffic lights (red, yellow, green) and their corresponding distances, can be received and saved first, and the crossing distance when the recognition result is a zebra crossing is recorded as d1, and the crossing distance when the recognition result is a traffic light is recorded as d2. Then the target crossing image is judged. When the target crossing image is a zebra crossing image, the crossing state judgment and distance information setting are performed to determine whether the crossing distance is greater than the preset road edge distance. The preset road edge distance can be set to 1.5m. If the crossing distance is greater than the preset road edge distance, that is, d1>1.5m, it can be determined that the user is approaching the road but has not yet started to cross the street, d1 is set as the distance information, and the voice information of not crossing the street is broadcasted through the voice module to prompt the user to move forward d1 meters to reach the edge of the road. Exemplarily, when the traffic light is red, the output voice information of not crossing the street can be "the current red light is d1 meters away from the road". If the crossing distance is not greater than the preset road edge distance, that is, d1≤1.5m, it can be determined that the user is crossing the street, and the voice information of crossing the street is broadcast through the voice module. Exemplarily, when the traffic light is green, the output voice information of crossing the street can be "the current light is green, the edge of the road has been reached, and the street can be crossed". When the target crossing image is a traffic light image, it can be determined that the user is about to arrive, and d2 can be set as distance information according to the crossing distance, and the voice information of about to pass can be broadcast through the voice module to prompt the user that it is still d2 meters away from crossing the road to reach the other side of the road. Exemplarily, when the traffic light is green, the output voice information of about to pass can be "the current light is green, and there are still d2 meters to cross the road". More, information processing and voice broadcasting can be performed by a microcontroller. The microcontroller first processes the received information and generates corresponding voice prompts for broadcasting. If it is other markers, the category and distance information of the traffic sign are provided to the user through the voice broadcast system to assist the blind in navigation and obstacle avoidance. This embodiment is more humanely designed and can intelligently process voice broadcast information in combination with scenarios from a practical perspective.

[0138] In some embodiments, in the safe crossing mode, the overall process of calculating the crossing distance is as follows: Figure 5 As shown, traffic signs can be recognized first. If a zebra crossing or a traffic light is recognized, monocular distance measurement is performed to record the zebra crossing distance d1, the color color and the traffic light distance d2, and then the distance information is judged. If the distance is less than 1.5m, the output distance information is the sidewalk, the traffic light color, and the distance d1; otherwise, the output distance information is the traffic light color and the distance d2.

[0139] In some embodiments, in step S107, the pattern recognition result may be judged, and when the pattern recognition result is a safe blind path pattern, steps S108 to S110 are executed.

[0140] In some embodiments, in step S108, the blind path image to be processed may be extracted from the environment image. The blind path image to be processed includes the blind path area. Figure 6 As shown, the recognition accuracy is 0.87. The extraction method may include but is not limited to image cropping, image enlargement or image reduction.

[0141] In some embodiments, in step S109, calculating the distance of obstacles in the blind path according to the blind path image to be processed may include but is not limited to the following steps:

[0142] Perform morphological processing on the blind path image to be processed to obtain a morphological image;

[0143] Perform color feature extraction on the morphological image to obtain the target blind path image;

[0144] Extracting a blind path region from the target blind path image, where the blind path region includes a normal blind path region or an abnormal blind path region;

[0145] Calculate the obstacle measurement width according to the blind path measurement width, the obstacle pixel width and the blind path pixel width of the blind path area;

[0146] Calculate the obstacle distance of the blind path based on the measured width of the obstacle.

[0147] In some embodiments, the morphological processing of the blind path image to be processed can be performed on the K210 platform to obtain a morphological image, and the color feature extraction processing can be performed on the morphological image to obtain a target blind path image, so as to facilitate more accurate calculations in the future. Then, the blind path area is extracted from the target blind path image, wherein the blind path area can include a normal blind path area or an abnormal blind path area, and the recognition module is called to determine whether the abnormal obstacle is identifiable. For the normal blind path area, the distance calculation can be performed directly through the ranging module. For the abnormal blind path area, the reference value is directly called for the identifiable target obstacle to use the ranging module to calculate the distance. For unidentified targets, let the pixel width of the blind path be recorded as P. A , find the widest obstacle pixel width in the unrecognized target in the blind path abnormal area, denoted as P B , the measured width of the blind path is recorded as W A , the measured width of the obstacle is recorded as W BThen, according to the measured width of the blind path, the pixel width of the obstacle and the pixel width of the blind path in the blind path area, the principle of similar triangles is applied to obtain the proportional relationship between the pixel width and the actual width, and the measured width of the obstacle is calculated. Among them, the calculation formula of the measured width of the obstacle is: Where W B Measure the width of the obstacle, P B is the pixel width of the obstacle, W A P is the measured width of the blind path. A is the pixel width of the blind path. The result of identifying the blind path obstacle is as follows Figure 7 As shown, the recognition accuracy is 0.93. It can be understood that when the blind path measurement width W is known A When the blind path pixel width P A and obstacle pixel width P B It can be obtained from the Pos array in the recognition module. Finally, the width is measured according to the obstacle, and W is B The information is transmitted to the abnormal position detection module, and the distance to the obstacle on the blind path is calculated by the distance measurement module, so that the blind can be reminded to avoid obstacles through the voice module later.

[0148] In some embodiments, in step S110, according to the distance of the obstacle on the blind path, the blind guide assistance is provided by the voice module, which may include but is not limited to the following steps:

[0149] When the blind path area is a normal blind path area, the normal blind path voice information is broadcast through the voice module;

[0150] When the blind path area is an abnormal blind path area, the abnormal blind path voice information is broadcast through the voice module according to the distance of the blind path obstacle.

[0151] In some embodiments, the blind path area can be judged first. When the blind path area is a normal blind path area, it can be judged that the blind path can be walked normally, and the normal voice information of the blind path is broadcasted through the voice module. Exemplarily, the normal voice information of the blind path can be "the blind path ahead is normal and can be walked". When the blind path area is an abnormal blind path area, it can be judged that an obstacle appears in the blind path and it is necessary to avoid the obstacle when walking. According to the distance of the blind path obstacle, the abnormal voice information of the blind path is broadcasted through the voice module. Exemplarily, the distance of the blind path obstacle is recorded as d3, and the abnormal voice information of the blind path can be "an obstacle appears at a distance of d3 meters from the blind path ahead, please avoid the obstacle when walking".

[0152] In some embodiments, in the safe blind path mode, the overall process of calculating the distance of the blind path obstacle is as follows: Figure 8As shown, the blind path image can be input first, color feature extraction and morphological processing can be performed, and then the blind path area can be extracted. If the blind path area is complete, the distance information is output; otherwise, it is determined whether the obstacle in the image is identifiable. If it is identifiable, the distance information is calculated by monocular ranging and output; otherwise, the width is estimated by the ratio of the obstacle to the blind path, and the distance information is calculated by monocular ranging and output.

[0153] In some embodiments, the overall process of performing blind guide assistance is as follows: Fig. 9 As shown, the visual recognition module can be implemented on the K210 platform and the traffic sign recognition model can be built. The traffic sign ranging module can be implemented on the K210 platform. Then, the image information sent by the camera can be received, and the traffic sign can be identified through the visual recognition module. The ranging module can be used to measure the distance of the identified information, and the category and distance information can be sent to the microcontroller. Finally, the microcontroller processes the information and sends it to the voice module for voice broadcast.

[0154] The beneficial effects of implementing the embodiments of the present invention include: the embodiments of the present invention first obtain an environmental image, perform pattern recognition processing through a traffic sign recognition model, and obtain a pattern recognition result. When the pattern recognition result is a safe street crossing mode, the target street crossing image is extracted from the environmental image, and the street crossing distance is calculated, and then the blind guide assistance is performed through a voice module, or, when the pattern recognition result is a safe blind road mode, the blind road image to be processed is extracted from the environmental image, and the distance of the blind road obstacle is calculated, and then the blind guide assistance is performed through a voice module, so that the environment can be perceived through image processing, and dual-mode blind guide assistance is achieved by identifying the safe street crossing mode or the safe blind road mode, thereby improving the environment perception ability and navigation ability. At the same time, this embodiment innovatively designs dual-mode blind guide assistance, namely the safe street crossing mode and the safe blind road mode, which not only takes into account the needs of the blind when crossing the road, but also takes into account the actual situation of walking on the blind road, more comprehensively meets the main travel scenarios of the blind, and provides accurate and detailed navigation information. In addition, this embodiment uses the K210 platform to achieve fast image processing and data transmission to ensure real-time information updates; through accurate distance calculation and anomaly detection, safety risks are effectively avoided; and navigation information is broadcast by voice to facilitate blind users to understand and operate. This embodiment is compact and portable, economical and practical, significantly improves durability, reduces the problem of fragility, and enables blind users to rely on it for a longer period of time. It also relies on advanced hardware to capture medium and long-distance environmental information, providing blind users with more comprehensive environmental perception, and is suitable for promotion and use.

[0155] like Fig.10 As shown, an embodiment of the present invention further provides a dual-mode blind guide assistance system, including:

[0156] An image acquisition module, which is used to acquire an environment image;

[0157] A scene judgment module is used to perform pattern recognition processing based on the environment image through the traffic sign recognition model to obtain a pattern recognition result, and the pattern recognition result includes a safe street crossing mode or a safe blind road mode;

[0158] Safe street crossing module, which is used to calculate the street crossing distance according to the safe street crossing mode;

[0159] Safety blind path module: The safety blind path module is used to calculate the distance of obstacles in the blind path according to the safety blind path mode;

[0160] The voice module is used to broadcast the position voice information according to the target distance information to achieve blind guide assistance. The target distance information includes the street crossing distance or the distance of the blind path obstacle.

[0161] In some embodiments, a dual-mode blind guide auxiliary system provided by an embodiment of the present invention includes an image acquisition module 201, a scene judgment module 202, a safe crossing module 203, a safe blind road module 204, and a voice module 205. The environmental image can be acquired through the image acquisition module 201. A camera device can be used for image acquisition. According to the environmental image, the scene judgment module 202 can be used to perform pattern recognition processing using a traffic sign recognition model to obtain a pattern recognition result. Among them, the pattern recognition result can include a safe crossing mode or a safe blind road mode. The scene judgment module is loaded with a visual recognition module based on the YOLOv2 algorithm. Among them, the visual recognition module is used to collect a traffic sign data set, including key traffic features such as traffic lights, zebra crossings, and blind roads, which are processed to improve the generalization ability of the model, and the traffic sign recognition model is trained based on the YOLOv2 algorithm using this traffic sign data set, and the traffic sign recognition model is built on the K210 platform. According to the safe crossing mode, the crossing distance can be calculated by the safe crossing module 203. The distance of obstacles in the blind path can be calculated by the safe blind path module 204 according to the safe blind path mode. The position voice information can be broadcasted by the voice module 205 according to the target distance information, that is, the received position information is broadcasted and voice navigation is performed, and the category and distance information of traffic signs are provided to the user to assist the blind in navigation and obstacle avoidance, and to achieve blind guide assistance. Among them, the target distance information may include the distance to cross the street or the distance to obstacles in the blind path. The voice module can be supported by an MP3-TF-16P audio device. Furthermore, the camera device or the K210 platform device of this embodiment can be worn on the chest of a blind user, and the camera device of the K210 platform can be used to collect images of the surrounding environment information.

[0162] In some embodiments, the safe crossing module includes:

[0163] A traffic sign recognition submodule, which is used to extract a target street crossing image from an environment image, wherein the target street crossing image includes a zebra crossing image or a traffic light image;

[0164] The monocular distance measurement submodule is used to calculate the distance of the identified object based on the target crossing street image;

[0165] The position information processing submodule is used to calculate the street crossing distance according to the distance of the identified object.

[0166] In some embodiments, the safe street crossing module may include a traffic sign recognition submodule 301, a monocular distance measurement submodule 302, and a position information processing submodule 303. The traffic sign recognition submodule 301 may be used to extract a target street crossing image from an environmental image. The target street crossing image may include a zebra crossing image or a traffic light image. The zebra crossing image information or the state of a traffic light may also be identified from the street crossing image. The distance of the identified object may be calculated by the monocular distance measurement submodule 302 according to the target street crossing image, that is, the distance calculation is performed on the identified traffic sign. The street crossing distance may be calculated by the position information processing submodule 303 according to the distance of the identified object, the distance information of the monocular distance measurement module is processed, and important position information is extracted and output to the voice module.

[0167] In some embodiments, the safety blind path module includes:

[0168] An image processing submodule is used to extract a to-be-processed blind path image from an environmental image and perform a preset image processing to obtain a target blind path image. The preset image processing includes a color feature extraction processing or a morphological processing;

[0169] A blind path extraction submodule is used to extract a blind path area from a target blind path image, where the blind path area includes a normal blind path area or an abnormal blind path area;

[0170] The abnormal position detection submodule is used to calculate the distance of the blind path obstacle according to the blind path area.

[0171] In some embodiments, the safe blind path module may include an image processing submodule 401, a blind path extraction submodule 402, and an abnormal position detection submodule 403. The image processing submodule 401 may be used to extract the blind path image to be processed from the environmental image, and perform preset image processing to obtain the target blind path image. The preset image processing may include color feature extraction processing or morphological processing. The blind path extraction submodule 402 may be used to extract the blind path area from the target blind path image. The blind path area may include a normal blind path area or an abnormal blind path area. According to the blind path area, the distance of the blind path obstacle may be calculated by the abnormal position detection submodule 403, that is, the distance calculation is performed for the abnormal blind path area, and the important position is output to the voice module.

[0172] The beneficial effects of implementing the embodiments of the present invention include: the embodiments of the present invention provide a dual-mode blind guide assistance system including an image acquisition module, a scene judgment module, a safe street crossing module, a safe blind path module, and a voice module. Among them, the image acquisition module can be used to obtain environmental images, the scene judgment module can be used to perform pattern recognition processing to obtain pattern recognition results, the safe street crossing module can be used to calculate the street crossing distance, the safe blind path module can be used to calculate the distance of obstacles on the blind path, and the voice module can be used to broadcast the position voice information, so that environmental perception can be performed through image processing, and dual-mode blind guide assistance can be achieved by identifying the safe street crossing module or the safe blind path module, thereby improving environmental perception and navigation capabilities.

[0173] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A dual-mode blind guide assistance method, It is characterized in that The following steps are involved: Get the environment image; According to the environmental image, pattern recognition processing is performed through a traffic sign recognition model to obtain a pattern recognition result, wherein the pattern recognition result includes a safe street crossing mode or a safe blind road mode; When the pattern recognition result is the safe street crossing mode, extracting a target street crossing image from the environment image, wherein the target street crossing image includes a zebra crossing image or a traffic light image; Calculating the street crossing distance according to the target street crossing image; According to the crossing distance, the blind guide is assisted by a voice module; or; When the pattern recognition result is the safe blind path pattern, extracting a to-be-processed blind path image from the environment image; Calculating the distance of obstacles on the blind path according to the blind path image to be processed; According to the distance of the obstacle on the blind path, guidance assistance is provided through a voice module.

2. The method according to claim 1, It is characterized in that The traffic sign recognition model is obtained by the following steps: obtaining a traffic image set; According to preset image requirements, the traffic image set is screened and processed; Performing training set division processing on the traffic image set after screening processing to obtain a training set to be processed; Performing noise reduction processing on the training set to be processed to obtain a first training set; Performing contrast optimization processing on the first training set to obtain a second training set; performing scale normalization processing on the second training set to obtain a third training set; Performing random rotation processing on the third training set to obtain a fourth training set; Performing random fuzzy processing on the fourth training set to obtain a fifth training set; Performing cropping and adjustment processing on the fifth training set to obtain a target training set; The target training set is input into a preset deep learning model so that the preset deep learning model is trained to obtain the traffic sign recognition model.

3. The method according to claim 1, It is characterized in that The method of performing pattern recognition processing according to the environment image by using a traffic sign recognition model to obtain a pattern recognition result includes: Inputting the environment image into the traffic sign recognition model for recognition to obtain a traffic sign recognition result; When the traffic sign recognition result is a street crossing sign, taking the safe street crossing mode as the pattern recognition result; When the traffic sign recognition result is a blind path sign, the safe blind path pattern is used as the pattern recognition result.

4. The method according to claim 1, It is characterized in that The calculating the street crossing distance according to the target street crossing image comprises: The monocular camera is calibrated by the four-point calibration method to obtain the camera parameter matrix; Constructing a ranging model according to the camera parameter matrix; Extracting a frame image and identifying a target frame from the target street crossing image; According to the recognition target frame, identifying the zebra crossing pixel width of the zebra crossing marker from the frame image; Calculating the focal length according to the distance measurement model, the object reference width, the reference distance and the zebra crossing pixel width; The street crossing distance is calculated according to the focal length, the object measurement width and the zebra crossing pixel width.

5. The method according to claim 1, It is characterized in that The providing blind-guiding assistance through a voice module according to the street crossing distance includes: When the target street crossing image is the zebra crossing image, determining whether the street crossing distance is greater than a preset road edge distance; If the crossing distance is greater than the preset road edge distance, the voice information of not crossing the street is broadcasted through the voice module; If the crossing distance is not greater than the preset road edge distance, the voice information of crossing the street is broadcasted through the voice module; When the target street crossing image is the traffic light image, the upcoming crossing voice information is announced through the voice module according to the street crossing distance.

6. The method according to claim 1, It is characterized in that The step of calculating the distance of obstacles on the blind path according to the blind path image to be processed includes: Performing morphological processing on the blind path image to be processed to obtain a morphological image; Performing color feature extraction processing on the morphological image to obtain a target blind path image; Extracting a blind path area from the target blind path image, wherein the blind path area includes a normal blind path area or an abnormal blind path area; Calculate the obstacle measurement width according to the blind path measurement width, the obstacle pixel width and the blind path pixel width of the blind path area; The obstacle distance of the blind path is calculated according to the measured width of the obstacle.

7. The method according to claim 6, It is characterized in that The method of providing blind-guiding assistance through a voice module according to the distance of the blind path obstacle includes: When the blind path area is the normal blind path area, the normal blind path voice information is broadcasted through the voice module; When the blind path area is the blind path abnormal area, the blind path abnormal voice information is broadcasted through the voice module according to the blind path obstacle distance.

8. A dual-mode blind guide assistance system, It is characterized in that include: An image acquisition module, wherein the image acquisition module is used to acquire an environment image; A scene judgment module, the scene judgment module is used to perform pattern recognition processing according to the environmental image through a traffic sign recognition model to obtain a pattern recognition result, wherein the pattern recognition result includes a safe street crossing mode or a safe blind road mode; A safe street crossing module, the safe street crossing module is used to calculate the street crossing distance according to the safe street crossing mode; A safety blind path module, the safety blind path module is used to calculate the distance of the blind path obstacle according to the safety blind path mode; A voice module is used to broadcast the position voice information according to the target distance information to achieve blind guide assistance, and the target distance information includes the street crossing distance or the blind path obstacle distance.

9. The system according to claim 8, It is characterized in that The safe crossing module comprises: A traffic sign recognition submodule, the traffic sign recognition submodule is used to extract a target street crossing image from the environment image, the target street crossing image includes a zebra crossing image or a traffic light image; A monocular distance measurement submodule, the monocular distance measurement submodule is used to calculate the distance of the identified object according to the target crossing street image; The position information processing submodule is used to calculate the street crossing distance according to the distance of the identification object.

10. The system according to claim 8, It is characterized in that The safety blind path module includes: An image processing submodule, the image processing submodule is used to extract the to-be-processed blind path image from the environment image, and perform preset image processing to obtain a target blind path image, wherein the preset image processing includes color feature extraction processing or morphological processing; A blind path extraction submodule, the blind path extraction submodule is used to extract a blind path area from the target blind path image, the blind path area includes a normal blind path area or an abnormal blind path area; The abnormal position detection submodule is used to calculate the distance of the obstacle of the blind path according to the blind path area.