Blind guiding equipment control method and related product

Abstract

The invention provides a blind guiding equipment control method and related products, and the method comprises the steps: obtaining a visual image and environment detection data of a reference region corresponding to a current position after a navigation route of a user from the current position to a destination is determined; fusing first feature data extracted from the visual image and second feature data extracted from the environment detection data to obtain a reference feature data set of the target object in the reference area; determining relative position parameters between all the target objects and the current position based on the reference feature data set; when the obstacle exists in the advancing direction of the user and the distance between the obstacle and the current position is smaller than or equal to a preset threshold value, updating the navigation route according to the relative position parameter corresponding to the obstacle, and generating a corresponding navigation control signal. Through implementation of the scheme of the invention, accurate and comprehensive road condition analysis can be realized, and safe and efficient obstacle avoidance guidance is provided for a user.

Description

Technical Field

[0001] The present application relates to the field of blind guide assistance technology, and in particular to a blind guide device control method and related products. Background Art

[0002] As a vulnerable group in society, visually impaired patients lack an important way to perceive the outside world and face various problems in their work and life. In related technologies, the mainstream way of guiding the blind is to set up blind paths and train guide dogs. However, the functions of existing blind paths are very limited, and users cannot accurately know the environmental information they are in based on the blind paths alone; in addition, the existing blind paths are all concave and convex tactile sensing blind paths, which require special floor tiles, are troublesome to construct, and are difficult to repair in time after damage; from the actual effect, the utilization rate of existing blind paths is also very low. The number of guide dogs is small, the training cost is high, and there is a lack of universality. The travel efficiency and sense of security of visually impaired patients need to be improved. Summary of the invention

[0003] The embodiments of the present application provide a method for controlling a guide device and related products, which can at least solve the problem of travel inconvenience for visually impaired patients in related technologies.

[0004] A first aspect of an embodiment of the present application provides a method for controlling a blind guide device, including:

[0005] After determining a navigation route for a user to reach a destination from a current location, obtaining a visual image and environmental detection data of a reference area corresponding to the current location; wherein a range of the reference area is determined based on the current location and the navigation route;

[0006] The first feature data extracted from the visual image and the second feature data extracted from the environmental detection data are fused to obtain a reference feature data set of the target object in the reference area; wherein the target object includes obstacles and non-obstacles;

[0007] Based on the reference feature data set, determine the relative position parameters between all target objects and the current position;

[0008] When it is determined based on all relative position parameters that the preset navigation control conditions are met, the navigation route is updated according to the relative position parameters corresponding to the obstacle, and a corresponding navigation control signal is generated; wherein the navigation control conditions include the presence of an obstacle in the user's direction of travel, and the distance between the obstacle and the current position is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to walk around avoiding obstacles.

[0009] A second aspect of an embodiment of the present application provides a blind guide device control device, including:

[0010] An acquisition module, configured to acquire a visual image and environmental detection data of a reference area corresponding to the current position after determining a navigation route for the user to reach a destination from the current position; wherein the range of the reference area is determined based on the current position and the navigation route;

[0011] A fusion module, used to fuse the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature data set of the target object in the reference area; wherein the target object includes obstacles and non-obstacles;

[0012] A determination module, used to determine relative position parameters between all target objects and the current position based on a reference feature data set;

[0013] A generation module is used to update the navigation route according to the relative position parameters corresponding to the obstacle and generate a corresponding navigation control signal when it is determined that the preset navigation control conditions are met based on all relative position parameters; wherein the navigation control conditions include the existence of an obstacle in the user's direction of travel and the distance between the obstacle and the current position is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to walk around the obstacle.

[0014] A third aspect of an embodiment of the present application provides a guide device for the blind, including: a guide instrument for the blind, a memory and a controller, wherein the guide instrument is configured with an audio output component, an image collector and a laser radar detector which are respectively connected to the controller for communication; the audio output component is used to output navigation information based on a navigation control signal sent by the controller, the image collector is used to collect visual images, and the laser radar detector is used to obtain environmental detection data; the controller is used to execute a computer program stored in the memory, and when the controller executes the computer program, the steps in the guide device control method provided in the first aspect of the embodiment of the present application are implemented.

[0015] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a controller, each step of the guide device control method provided in the first aspect of the embodiment of the present application is implemented.

[0016] As can be seen from the above, according to the guide device control method and related products provided by the present application scheme, after determining the navigation route of the user from the current position to the destination, the visual image and environmental detection data of the reference area corresponding to the current position are obtained; wherein the range of the reference area is determined based on the current position and the navigation route; the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data are fused to obtain a reference feature data set of the target object in the reference area; wherein the target object includes obstacles and non-obstacles; based on the reference feature data set, the relative position parameters between all target objects and the current position are determined; when it is determined based on all relative position parameters that the preset navigation control conditions are met, the navigation route is updated according to the relative position parameters corresponding to the obstacle, and a corresponding navigation control signal is generated; wherein the navigation control conditions include the presence of an obstacle in the direction of the user's travel, and the distance between the obstacle and the current position is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to walk around the obstacle. Through the implementation of the solution of the present application, multiple sensors are used to obtain road condition information of the reference area corresponding to the user's current location, that is, the visual images and environmental detection data collected by multiple sensors are fused and processed, so as to accurately locate the position information of obstacles, improve the accuracy, robustness and adaptability of obstacle detection, and significantly improve the safety and convenience of users walking in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A basic flow chart of a method for controlling a guide device for the blind provided in the first embodiment of the present application;

[0018] Figure 2 A schematic diagram of the structure of a first-view guide device for the blind provided in the first embodiment of the present application;

[0019] Figure 3 A schematic diagram of the structure of a second viewing angle blind guide provided in the first embodiment of the present application;

[0020] Figure 4 A schematic structural diagram of a touch guide provided in the first embodiment of the present application;

[0021] Figure 5 A detailed flowchart of a method for controlling a guide device for the blind provided in the second embodiment of the present application;

[0022] Figure 6 A schematic diagram of a program module of a blind guide device control device provided in the third embodiment of the present application;

[0023] Figure 7 This is a schematic diagram of the structure of a guide device for the blind provided in the fourth embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0025] In the description of the embodiments of the present application, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0026] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0027] In the embodiments of the present application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0028] In order to solve the problem of inconvenience in travel for visually impaired patients in the related art, the first embodiment of the present application provides a method for controlling a guide device, such as Figure 1 A basic flow chart of the blind guide device control method provided in this embodiment, the blind guide device control method comprises the following steps:

[0029] Step 101: After determining a navigation route for a user to reach a destination from a current location, obtain a visual image and environmental detection data of a reference area corresponding to the current location.

[0030] Specifically, the range of the reference area is determined based on the current location and the navigation route. Figure 2 and 3 As shown, in this embodiment, the blind guide device can be configured with a small and lightweight blind guide 100, and an image collector 1, a laser radar detector 2, and an ultrasonic detector 3 can be integrated on the blind guide 100 to implement the acquisition of road condition information; the blind guide 100 is also provided with a wearing component (not shown in the figure), which is convenient for users to install it on helmets, blind guide sticks, shoulders, chests, waists, etc. according to personal habits and needs. The image collector 1 may include a night camera 11 and a daytime camera 12. In addition, a keyboard 4, a display 5, a wireless charging induction component 6, etc. can also be set on the blind guide 100, wherein the keyboard 4 can be used to set parameters and function debugging, and the display can be used to display relevant information to cooperate with the completion of function debugging. The intelligent blind guide 100 of this embodiment can be powered by a rechargeable battery to ensure long-term use. At the same time, it can be equipped with a low-battery reminder function, and when the battery is low, it can remind the user to charge in time; considering the user's convenience, the blind guide 100 is also provided with a wireless charging induction component 6, which supports wireless charging technology and can be used for wireless charging to facilitate users to charge at any time when going out.

[0031] In some specific embodiments, the image collector 1 can be a 180-degree wide-angle camera, and the visual image captured by the 180-degree wide-angle camera can be used to analyze the road conditions, such as obstacles, puddles, steps, etc., and then the visual image can be identified and analyzed by the image processing algorithm to determine the best walking path; the laser radar detector 2 and the ultrasonic detector 3 can be used to measure the distance and position of the obstacles in front and around in real time, provide accurate spatial information for navigation, and ensure the comprehensiveness, accuracy and real-time nature of data collection. In a specific embodiment, the above-mentioned environmental detection data can refer to the three-dimensional point cloud data of the laser radar detector 2, that is, the laser radar detector 2 can obtain the three-dimensional point cloud data of the user's surrounding environment by emitting a laser beam and receiving a reflected signal; the three-dimensional point cloud data includes information such as the distance between the target object and the user, the shape and position of the target object.

[0032] It can be understood that in some other embodiments, the image collector 1 can be a visible light camera, an infrared camera, a depth camera, etc., which is not limited here; different sensors have their own advantages in different regions and weather conditions; for example, in low light or night environments, infrared cameras can capture the thermal radiation of objects to help identify the contours of objects; depth cameras can provide distance information of objects, which helps to determine the location and distance of obstacles.

[0033] It should be noted that, in some embodiments, the navigation route determined by the guide device 100 for the blind can be recorded in advance, or can be determined by loading a map navigation. That is, the guide device 100 for the blind can record in advance the designated routes that the user often passes, such as commuting routes, routes to the park, routes to the supermarket, etc. When the user wants to go to a place that he or she does not often go to or a remote place, the map navigation can be loaded, and the navigation route can be updated according to the real-time traffic information. The guide device 100 of this embodiment can be achieved without completely relying on the map, and can replace the blind person's eyes to observe the road conditions, and can also identify roadblocks, road signals and instructions, etc., so that the user can walk on the road easily and safely.

[0034] Step 102: Fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature data set of the target object in the reference area.

[0035] Specifically, the target object includes obstacles and non-obstacles, and the target object can be a static object or a dynamic object. Before data fusion, the above-mentioned visual image can be subjected to pre-processing operations such as denoising, enhancement (such as brightness and contrast adjustment) and edge detection to obtain optimized two-dimensional image data (that is, the above-mentioned first feature data) to improve the accuracy of subsequent recognition. The environmental detection data obtained by the laser radar detector 2 can be filtered, downsampled, aligned (when multiple radar detectors are used to obtain detection data) and segmented to extract useful second feature data. Through the fusion processing of the first feature data and the second feature data, a more comprehensive and real-time perception and intelligent analysis of the surrounding environment can be achieved.

[0036] Step 103: Determine relative position parameters between all target objects and the current position based on the reference feature data set.

[0037] Specifically, trained deep learning models (such as convolutional neural networks) can be used to detect target objects such as pedestrians, vehicles, and roadblocks. Geometric feature analysis combined with machine learning methods can also be used to extract target objects from point clouds.

[0038] Step 104: When it is determined based on all relative position parameters that a preset navigation control condition is satisfied, the navigation route is updated according to the relative position parameters corresponding to the obstacle, and a corresponding navigation control signal is generated.

[0039] Specifically, the navigation control conditions include the presence of an obstacle in the direction of the user's travel, and the distance between the obstacle and the current position is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to avoid obstacles. Figure 2As shown, the guide device 100 of this embodiment can also be configured with an audio output component (including a speaker 7). In addition, for users with hearing impairments or who do not want to disturb others, the following can also be configured: Figure 4 The tactile guide 200 shown in the figure can be provided with a vibration straightener 8 and a braille display component 9. Accordingly, the navigation control signal of the present embodiment can refer to the control signal of the audio output component, can also refer to the control signal of the vibration straightener 8, and can also refer to the control signal of the braille display component 9. The audio output component can clearly and accurately voice prompt the user of the road conditions ahead, the location of obstacles and the recommended walking direction, so as to ensure that the user can obtain navigation information in time. The vibration straightener 8 can generate rolling vibrations in different directions, such as forward, backward, left and right turns, turning around, etc., to realize tactile navigation, so that the user can perceive the guiding direction through touch. The braille display component 9 can convert the text navigation information into braille form, so that the blind or visually impaired can touch and read to realize direction guidance. In the actual application process, the user can select the desired signal output mode according to actual needs, which is not limited here.

[0040] In some implementations of this embodiment, when determining the navigation route, the road condition information can be determined based on a preset path search algorithm to search for the best walking path (i.e., the navigation route). Specifically, the nodes and edges in the graph structure are first determined, the nodes can be used to represent the position, and the edges can be used to represent the walking direction and distance; the estimated distance from each node to the target location and the actual distance traveled are calculated based on the preset path search algorithm, and then the optimal navigation route can be determined. Optionally, the above-mentioned path search algorithm can be an A* algorithm, a Dijkstra algorithm, etc.

[0041] In some implementations of this embodiment, the parameters of the path search algorithm can also be adjusted according to different terrain conditions and weather conditions. For example, in mountainous areas, due to the complex terrain, the safety weight of the path can be increased to avoid choosing a path that is too steep or dangerous. On rainy days, due to the slippery roads, the stability weight of the path can be increased to avoid choosing a path that is easy to slip.

[0042] In some specific implementations, the route planning can also be optimized in combination with user feedback information. For example, if a user feedbacks that a certain route is too crowded or unsafe, the parameters of the route search algorithm can be adjusted to avoid selecting that route. At the same time, personalized route planning services can be provided based on user preferences, such as giving priority to the shortest route, the fastest route, the route with beautiful scenery, etc.

[0043] In some implementations of this embodiment, when determining a specific walking path, a machine learning algorithm suitable for road condition recognition and path planning can be selected. For example, a convolutional neural network (CNN) can be used to extract feature data and detect targets from visual images, that is, a large amount of labeled image data can be used to train a convolutional neural network model. The accuracy and robustness of the convolutional neural network can be improved by adjusting the network structure, hyperparameters, etc. During the training process, data enhancement techniques such as random rotation, flipping, scaling, etc. can be used to increase the diversity of the data and prevent overfitting.

[0044] In some implementations of this embodiment, a reinforcement learning algorithm can also be combined to carry out specific walking path planning. Specifically, based on the preset reinforcement learning algorithm, a suitable state space, action space and reward function can be designed; wherein the state space can include information such as the current position, image features of the surrounding environment, and the target position; the action space can include walking directions such as forward, backward, left turn, and right turn; the reward function can be designed based on factors such as the length of the walking path, safety, and proximity to the target; the agent is allowed to learn in a simulated environment and continuously adjust the strategy to find the best walking path.

[0045] In some implementations of the present embodiment, before the above-mentioned step of obtaining the visual image and environmental detection data of the reference area corresponding to the current location, it also includes: after determining the user's destination, determining the user's item carrying information, and obtaining all optional routes for the user to reach the destination from the current location and the road condition information of each optional route; wherein the content of the road condition information includes terrain information, crowd density and weather conditions; based on the item carrying information and the road condition information, determining the navigation route for the user to reach the destination from the current location from all optional routes.

[0046] Specifically, the weight coefficients of various factors corresponding to the road condition information and the weight coefficients corresponding to the item carrying information can be set, and the priorities of various optional routes can be determined in combination with the actual road condition information and the item carrying status of the user, and then the priorities of various optional routes can be compared to select the best navigation route. For example, when the user carries items such as a suitcase, an optional route with a shorter walking time and no steps or fewer steps can be preferentially selected as the navigation route.

[0047] In some implementations of this embodiment, when preprocessing the above-mentioned visual images, different algorithm models can be used to perform preprocessing operations on visual images captured under different terrain conditions and weather conditions. For example, in hot areas, there may be strong direct sunlight, causing the image to be too bright; an adaptive brightness adjustment algorithm can be used to reduce the overall brightness of the image while enhancing the contrast to make the object clearer. In cold areas, there may be snow cover that affects the recognition of roads and objects; a snow removal algorithm can be used to remove the interference of snow by analyzing the color and texture features in the image. On rainy days, the image may be blurred by raindrops; a rain removal algorithm can be applied to remove the influence of raindrops on the image by detecting the shape and movement trajectory of raindrops. On foggy days, the clarity of the image will be reduced; a defogging algorithm, such as a defogging method based on dark channel prior, is used to restore the contrast and details of the image.

[0048] In some implementations of this embodiment, before the above-mentioned step of fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature data set of the target object in the reference area, potential target objects (such as pedestrians, vehicles, obstacles, etc.) can be detected and identified from the visual image to obtain the first feature data; specifically, a deep learning model (such as YOLO, Faster R-CNN, etc.) can be used for target detection, and when training the target detection model, an image data set containing different terrain conditions and weather conditions is used to improve the generalization ability of the model. The geometric characteristics (such as position, speed, etc.) of each detected target are extracted from the environmental detection data to obtain the second feature data.

[0049] Furthermore, in some specific implementations, when performing the extraction of the first feature data and target detection, a feature extraction method that is adapted to different terrain conditions and weather conditions can be designed. For example: in urban areas, there are many buildings and traffic signs, and geometric features such as edges and corners, as well as features such as color and texture can be extracted to identify targets such as buildings, traffic signs and vehicles. In mountainous areas, the terrain is complex, and features such as mountain contours and rivers can be extracted to determine the direction of the road and the location of obstacles. Under different weather conditions, adjust the parameters of feature extraction; for example: on rainy days, due to the reflection of rain, the color and texture of objects may change, and the method for extracting color and texture features can be adjusted.

[0050] In some implementations of the present embodiment, the above-mentioned step of fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature data set of the target object in the reference area includes: based on a preset timestamp, aligning the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data; after the data are aligned, associating the first feature data with the geometric features of the target object determined based on the second feature data to obtain association information; wherein the target object is an object in the reference area; based on the association information, fusing the first feature data and the second feature data to obtain a reference feature data set of the target object.

[0051] Specifically, in this embodiment, when the geometric features of the first feature data and the second feature data are associated, for each target object detected by the laser radar detector, the spatial distance between it and the target object detected in the visual image is calculated; the Euclidean distance or other metrics can be used to determine the degree of match between the two (set threshold); if an object matches the position of an object in the visual image within a certain spatial error range (the degree of match is greater than the set threshold), the two are associated to obtain corresponding association information. In addition, in some embodiments, for complex scenes, when there are multiple target objects, a joint probability data association method can be used to comprehensively consider the relationship between each detected target object to achieve more accurate information association.

[0052] Furthermore, in some implementations of the present embodiment, the above-mentioned step of fusing the first feature data and the second feature data based on the association information to obtain a reference feature data set of the target object includes: inputting the first feature data into a trained neural network model to obtain a reference image with the target object identification; based on the association information, mapping the second feature data to the reference image to generate a reference feature data set of the target object.

[0053] Specifically, in this embodiment, weighted averaging, feature-based fusion, particle filtering, etc. can be used to fuse the associated visual image and environmental detection data to obtain a reference feature data set of the target object; wherein the content of the reference feature data set includes the final position, category and other relevant information of each target object in space.

[0054] In some other implementations of this embodiment, the first feature data and the second feature data can be spliced ​​or weighted averaged at the same level to obtain fused feature data; the fused feature data is input into the trained deep learning model, and then the required reference feature data set can be output. Specifically, when constructing a deep learning model, the output layer architecture such as target classification and detection frame regression can be set according to actual needs; road image data and detection data under different terrain conditions and weather conditions are widely collected, and accurately annotated, and the annotated content can include the location and category of target objects such as roads, obstacles, and traffic signs, to ensure that the data set covers various terrain features (such as cities, villages, mountainous areas, etc.) and weather conditions (such as sunny days, rainy days, snowy days, foggy days, etc.); the data set is divided into a training set, a validation set, and a test set for model training, adjustment, and evaluation; during the training process, multiple iterations of training can be performed on the training data containing visual image features and lidar detection data features, and the loss value and accuracy during the training process can be monitored. Methods such as early stopping and cross-validation can also be used to prevent overfitting, and the model performance is evaluated in the validation set.

[0055] In some implementations of this embodiment, the deep learning model can be trained in combination with transfer learning and incremental learning methods so that the model can quickly adapt to new geographical and weather conditions. Transfer learning can apply a model trained under one terrain condition and weather condition to other similar terrain conditions and weather conditions, and perform fine-tuning to improve the adaptability of the model. Incremental learning allows the model to learn on new data, continuously update and optimize model parameters without retraining the entire model. In actual application, the model can be evaluated and updated regularly, and the model can be adjusted and optimized based on feedback information from actual navigation and newly collected data. For example, if it is found that the performance of the model has degraded under certain terrain conditions and weather conditions, the data under those conditions can be re-collected for training to improve the accuracy of the model.

[0056] In some implementations of the present embodiment, after the above-mentioned step of fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature data set of the target objects in the reference area, it also includes: establishing a three-dimensional motion model of each target object based on the environmental detection data; determining the movement trajectory of each target object within a preset time period based on the three-dimensional motion model; adding the position features of the navigation route in the reference area to the three-dimensional motion model to determine whether there is an intersection or overlap between each movement trajectory and the navigation route within the preset time; if the judgment result is yes, updating the navigation route based on each movement trajectory, and generating a corresponding navigation control signal.

[0057] Specifically, the target object of this embodiment can be a dynamic object or a static object. When performing feature extraction and target detection, a Kalman filter can be used to estimate the position and speed of the target object to help deal with noise and uncertainty, and generate a corresponding three-dimensional motion model, which can be used to predict the future position of the target object. Therefore, the three-dimensional motion model and the navigation route can be combined to determine whether the user's travel process will be hindered by obstacles within a preset time period in the future; if there is an obstacle, the navigation route is updated in real time and a corresponding navigation control signal is generated to guide the user to reach the destination smoothly. The guide device of this embodiment can control various sensors to continuously obtain new road condition information during the actual walking process and adjust the walking path in real time; if new obstacles or road condition changes are found, the path is replanned in time to ensure safety and efficiency.

[0058] In some implementations of this embodiment, the above-mentioned control method also includes: when the rescue function of the guide device is triggered, obtaining the user's distress information; wherein the distress information includes at least one of the user's vital signs information and the surrounding environment safety information of the current location; determining the target rescue platform based on the distress information, and sending a rescue request to the target rescue platform; wherein the rescue request carries the distress information and the positioning information of the guide device.

[0059] Specifically, the guide device of this embodiment can be configured with an SOS alarm function component, and an SOS one-key alarm button 7 is provided on the guide device 100. When the user presses the alarm button 7 in an emergency, the alarm process of the rescue function is immediately triggered. The intelligent guide device 100 can automatically obtain the user's current location information and send it to the corresponding SOS system through the built-in GPS positioning system. The guide device 100 can also link the camera and the sound receiving system thereon to capture the scene image and sound, and send the data together to the SOS system to provide more comprehensive information support for rescue; according to the received positioning data and on-site information, the target rescue platform can be quickly analyzed and determined, and then the target rescue platform can be notified to carry out timely and accurate rescue operations. In this embodiment, the target rescue platform can be public security, hospital, fire protection, etc., which is not limited here. It should be noted that in some embodiments, the target rescue platform can be determined by the evaluation model preset in the SOS system based on the received positioning data and on-site information, or the target rescue platform can be determined by the staff of the SOS system based on the received positioning data and on-site information, which is not limited here.

[0060] In actual application, the wearing method and sensor configuration of the smart guide 100 can be adjusted according to user needs and scene characteristics. The algorithm is optimized to adapt to navigation needs in different regions and weather conditions to ensure the practicality and reliability of the guide 100. At the same time, a matching mobile phone APP or remote controller can be developed to facilitate users to view historical walking records, adjust navigation settings, receive emergency notifications and perform other related operations. In addition, it is also possible to consider adding user-defined functions, such as setting emergency contacts, adjusting alarm volume and vibration intensity, etc., to meet the personalized needs of different users.

[0061] Based on the technical solution of the embodiment of the present application, it is possible to combine multi-sensor fusion technology, intelligent algorithms, multiple navigation prompt systems, emergency rescue functions (including SOS alarm button 7, positioning data transmission, camera and voice system linkage, etc.) and supporting APP (or remote controller) to achieve blind guidance. It is a highly integrated intelligent technology and humanized design, which can significantly improve the walking safety and convenience of users in complex environments; it is an important innovative achievement of modern technology to assist the lives of disabled people, and has broad application prospects and social value. It can fully consider the characteristics of different regions and weather conditions, combine multi-sensor data fusion, specific image preprocessing, feature extraction and target detection adapted to different conditions, machine learning algorithms and other means to optimize the guidance function of the blind guide equipment, which is conducive to improving the accuracy, robustness and adaptability of navigation route planning.

[0062] Figure 5 The method in is a detailed blind guide device control method provided in the second embodiment of the present application, and the blind guide device control method includes:

[0063] Step 501: After determining the navigation route of the user from the current location to the destination, obtain the visual image and environment detection data of the reference area corresponding to the current location.

[0064] Specifically, in this embodiment, the range of the reference area is determined based on the current location and the navigation route.

[0065] Step 502: Based on a preset timestamp, align the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data.

[0066] Step 503: After the data is aligned, the first feature data is associated with the geometric features of the target object determined based on the second feature data to obtain association information.

[0067] Specifically, the target object is an object in the reference area.

[0068] Step 504: Based on the association information, the first feature data and the second feature data are fused to obtain a reference feature data set of the target object.

[0069] Step 505: Determine relative position parameters between all target objects and the current position based on the reference feature data set.

[0070] Step 506: When it is determined based on all relative position parameters that there is an obstacle in the user's direction of travel and the distance between the obstacle and the current position is less than or equal to a preset threshold, the navigation route is updated according to the relative position parameters corresponding to the obstacle, and a corresponding navigation control signal is generated.

[0071] Specifically, the navigation control signal is used to control the guide device to output navigation information, and the navigation information is used to guide the user to walk around obstacles.

[0072] Through the implementation of this embodiment, data collected by multiple sensors (visible light cameras, infrared cameras, depth cameras, etc.) are combined to perform road condition analysis, and images from different sensors are fused using a data fusion algorithm, which is beneficial to improving the accuracy and robustness of road condition recognition, and realizing comprehensive and real-time perception and intelligent analysis of the surrounding environment, greatly improving the user's walking safety and autonomy.

[0073] It should be understood that the size of the serial number of each step in this embodiment does not mean the order of execution of the steps. The execution order of each step should be determined by its function and internal logic, and should not constitute a sole limitation on the implementation process of the embodiment of this application.

[0074] Figure 6 A blind guide device control device is provided in the third embodiment of the present application. The blind guide device control device can be applied to the aforementioned blind guide device control method. Figure 7 As shown, the blind guide equipment control device mainly includes:

[0075] The acquisition module 601 is used to acquire the visual image and environmental detection data of the reference area corresponding to the current position after determining the navigation route of the user from the current position to the destination; wherein the range of the reference area is determined based on the current position and the navigation route;

[0076] A fusion module 602 is used to fuse the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature data set of the target object in the reference area; wherein the target object includes obstacles and non-obstacles;

[0077] A determination module 603 is used to determine relative position parameters between all target objects and the current position based on the reference feature data set;

[0078] The generation module 604 is used to update the navigation route according to the relative position parameters corresponding to the obstacle and generate a corresponding navigation control signal when it is determined that the preset navigation control conditions are met based on all relative position parameters; wherein the navigation control conditions include the existence of an obstacle in the user's direction of travel and the distance between the obstacle and the current position is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to walk around the obstacle.

[0079] In some implementations of this embodiment, a navigation module is also included, and the navigation module is specifically used to: after determining the user's destination, determine the user's item carrying information, and obtain all optional routes for the user to reach the destination from the current location and the road condition information of each optional route; wherein the content of the road condition information includes terrain information, crowd density and weather conditions; based on the item carrying information and the road condition information, determine the navigation route for the user to reach the destination from the current location from all optional routes.

[0080] In some implementations of this embodiment, the fusion module is specifically used to: align the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data based on a preset timestamp; after the data are aligned, associate the first feature data with the geometric features of the target object determined based on the second feature data to obtain association information; wherein the target object is an object in a reference area; based on the association information, fuse the first feature data and the second feature data to obtain a reference feature data set of the target object.

[0081] Furthermore, in some implementations of the present embodiment, when the fusion module performs the above-mentioned function of fusing the first feature data and the second feature data based on the association information to obtain a reference feature data set of the target object, it is specifically used to: input the first feature data into the trained neural network model to obtain a reference image with the target object identification; based on the association information, map the second feature data to the reference image to generate a reference feature data set of the target object.

[0082] In some implementations of this embodiment, the navigation module is also specifically used to: establish a three-dimensional motion model of each target object based on environmental detection data; determine the movement trajectory of each target object within a preset time period based on the three-dimensional motion model; add the position features of the navigation route in the reference area to the three-dimensional motion model to determine whether there is an intersection or overlap between each movement trajectory and the navigation route within the preset time; if the judgment result is yes, update the navigation route based on each movement trajectory.

[0083] In some implementations of this embodiment, a rescue module is also included, which is specifically used to: when the rescue function of the guide device is triggered, obtain the user's distress information; wherein the distress information includes at least one of the user's vital signs information and the surrounding environment safety information of the current location; determine the target rescue platform based on the distress information, and send a rescue request to the target rescue platform; wherein the rescue request carries the distress information and the positioning information of the guide device.

[0084] According to the guide device control device provided by the present embodiment, after determining the navigation route of the user from the current position to the destination, the visual image and environmental detection data of the reference area corresponding to the current position are obtained; wherein the range of the reference area is determined based on the current position and the navigation route; the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data are fused to obtain a reference feature data set of the target objects in the reference area; wherein the target objects include obstacles and non-obstacles; based on the reference feature data set, the relative position parameters between all target objects and the current position are determined; when it is determined based on all relative position parameters that the preset navigation control conditions are met, the navigation route is updated according to the relative position parameters corresponding to the obstacle, and a corresponding navigation control signal is generated; wherein the navigation control conditions include the existence of an obstacle in the direction of the user's travel, and the distance between the obstacle and the current position is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to walk around the obstacle. Through the implementation of the solution of the present application, multiple sensors are used to obtain road condition information of the reference area corresponding to the user's current location, that is, the visual images and environmental detection data collected by multiple sensors are fused and processed, so as to accurately locate the position information of obstacles, improve the accuracy, robustness and adaptability of obstacle detection, and significantly improve the safety and convenience of users walking in complex environments.

[0085] Figure 7 A blind guide device is provided in the fourth embodiment of the present application. The blind guide device can be used to implement the blind guide device control method in the above embodiments, mainly including:

[0086] A memory 701, a controller 702, a computer program 703 stored in the memory 701 and executable on the controller 702, and a guide for the blind 704; the memory 701 and the controller 702 are connected by communication, and the guide for the blind 704 is configured with an audio output component, an image collector, and a laser radar detector respectively connected to the controller by communication; the audio output component is used to output navigation information based on a navigation control signal sent by the controller, the image collector is used to collect visual images, and the laser radar detector is used to obtain environmental detection data. When the controller 702 executes the computer program 703, the method in the first or second embodiment is implemented. The number of controllers may be one or more.

[0087] In this embodiment, a tactile guide is also configured to be communicatively connected to the controller, and the tactile guide includes a vibration straightener and / or a Braille display component for outputting navigation information based on the navigation control signal sent by the controller. It can be understood that in some other embodiments, the guide device and the tactile guide can also be integrated into a unified device, which is not limited here.

[0088] The memory 701 may be a high-speed random access memory (RAM) memory, or a non-volatile memory, such as a disk memory. The memory 701 is used to store executable program codes, and the controller 702 is coupled to the memory 701 .

[0089] Furthermore, the embodiment of the present application also provides a computer-readable storage medium, which may be provided in the above-mentioned blind guide device, and the computer-readable storage medium may be the above-mentioned Figure 7 Memory in the illustrated embodiment.

[0090] The computer readable storage medium stores a computer program, and when the program is executed by the controller, the control method of the guide device for the blind in the above-mentioned embodiment is implemented. Furthermore, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, etc., which can store program codes.

[0091] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0092] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0094] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a readable storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned readable storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0095] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0096] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] The above is a description of the guide device control method and related products provided by the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for controlling a blind guide device, characterized in that: include: After determining a navigation route for a user to reach a destination from a current location, obtaining a visual image and environmental detection data of a reference area corresponding to the current location; wherein a range of the reference area is determined based on the current location and the navigation route; Fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature data set of the target object in the reference area; wherein the target object includes obstacles and non-obstacles; Based on the reference feature data set, determining relative position parameters between all the target objects and the current position; When it is determined based on all the relative position parameters that a preset navigation control condition is met, the navigation route is updated according to the relative position parameter corresponding to the obstacle, and a corresponding navigation control signal is generated; wherein the navigation control condition includes that the obstacle exists in the direction of user travel, and the distance between the obstacle and the current position is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to walk away from the obstacle.

2. The method for controlling a blind guide device according to claim 1, characterized in that: Before acquiring the visual image and environment detection data of the reference area corresponding to the current position, the method further includes: After determining the destination of the user, determine the item carried by the user, and obtain all optional routes from the user's current location to the destination and the road condition information of each optional route; wherein the road condition information includes terrain information, crowd density and weather conditions; Based on the item carrying information and the road condition information, a navigation route for the user to reach a destination from a current location is determined from all the optional routes.

3. The method for controlling a blind guide device according to claim 1, characterized in that: The fusing of the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature data set of the target object in the reference area includes: Based on a preset timestamp, aligning the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data; After the data is aligned, the first feature data is associated with the geometric features of the target object determined based on the second feature data to obtain association information; wherein the target object is an object in the reference area; Based on the association information, the first feature data and the second feature data are fused to obtain a reference feature data set of the target object.

4. The method for controlling a blind guide device according to claim 3, characterized in that: The fusing the first feature data and the second feature data based on the association information to obtain a reference feature data set of the target object includes: Inputting the first feature data into the trained neural network model to obtain a reference image with a target object identification; Based on the association information, the second feature data is mapped to the reference image to generate a reference feature data set of the target object.

5. The method for controlling a blind guide device according to claim 1, characterized in that: After fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature data set of the target object in the reference area, the method further includes: Based on the environmental detection data, establishing a three-dimensional motion model of each of the target objects; Determine the movement trajectory of each of the target objects within a preset time period based on the three-dimensional motion model; Adding the position features of the navigation route in the reference area to the three-dimensional motion model, and determining whether there is an intersection or overlap between each of the movement trajectories and the navigation route within a preset time; If the judgment result is yes, the navigation route is updated based on each of the movement trajectories, and a corresponding navigation control signal is generated.

6. The method for controlling a blind guide device according to any one of claims 1 to 5, characterized in that: Also includes: When the rescue function of the guide device is triggered, obtaining the user's distress information; wherein the distress information includes at least one of the user's vital sign information and the safety information of the surrounding environment of the current location; A target rescue platform is determined based on the distress information, and a rescue request is sent to the target rescue platform; wherein the rescue request carries the distress information and the positioning information of the guide device.

7. A method for controlling a guide device for the blind, characterized in that: include: an acquisition module, configured to acquire a visual image and environmental detection data of a reference area corresponding to the current location after determining a navigation route for the user to reach a destination from the current location; wherein a range of the reference area is determined based on the current location and the navigation route; A fusion module, used for fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature data set of the target object in the reference area; wherein the target object includes obstacles and non-obstacles; A determination module, configured to determine relative position parameters between all the target objects and the current position based on the reference feature data set; A generation module is used to update the navigation route according to the relative position parameters corresponding to the obstacle and generate a corresponding navigation control signal when it is determined that a preset navigation control condition is met based on all the relative position parameters; wherein the navigation control condition includes that the obstacle exists in the user's direction of travel and the distance between the obstacle and the current position is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to walk around avoiding the obstacle.

8. A blind guide device, characterized in that: It includes a guide device, a memory and a controller, wherein: The guide device is configured with an audio output component, an image collector and a laser radar detector respectively connected to the controller for communication; the audio output component is used to output navigation information based on the navigation control signal sent by the controller, the image collector is used to collect visual images, and the laser radar detector is used to obtain environmental detection data; The controller is used to execute the computer program stored in the memory; When the controller executes the computer program, the steps in the method according to any one of claims 1 to 6 are implemented.

9. The blind guiding device according to claim 8, characterized in that: Also included is a tactile guide communicatively connected to the controller, the tactile guide including a vibrating hair straightener and / or a Braille display component for outputting navigation information based on a navigation control signal sent by the controller.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a controller, the steps in the method according to any one of claims 1 to 6 are implemented.

Images