Intelligent blind guiding method and device, electronic equipment and storage medium

Abstract

The invention provides an intelligent blind guiding method and device, electronic equipment and a storage medium, and relates to the computer technology, in particular to the technical field of intelligent blind guiding. According to the specific implementation scheme, the method comprises the steps of obtaining multi-modal sensing information; the multi-mode sensing information comprises images, videos, audios and pose information; preprocessing and fusing the multi-modal sensing information to obtain fused information; identifying the fusion information to obtain an identification result; the identification result comprises at least one of the following information: user surrounding environment information and user motion state information; and generating feedback information based on the identification result.

Description

Technical Field

[0001] The present disclosure relates to computer technology, and in particular to the field of intelligent blind-guiding technology, and specifically to an intelligent blind-guiding method, device, electronic device and storage medium. Background Art

[0002] Visually impaired people are those who are blind in one or both eyes due to diseases of the optic nerve or eyes or accidental injuries. Visually impaired people face difficulties and challenges in their daily lives due to visual impairment. With the development of science and technology, providing assistive technology for the visually impaired has become an important research direction. Summary of the invention

[0003] The present invention provides an intelligent blind-guiding method, device, electronic device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided an intelligent blind guiding method, comprising:

[0005] Acquire multimodal perception information; the multimodal perception information includes: image, video, audio and posture information;

[0006] Preprocessing and fusing the multimodal sensing information to obtain fused information;

[0007] Identify the fused information to obtain an identification result; the identification result includes at least one of the following: user surrounding environment information and user motion state information;

[0008] Feedback information is generated based on the recognition result.

[0009] According to another aspect of the present disclosure, there is provided an intelligent blind guiding device, comprising:

[0010] An acquisition unit, used to acquire multimodal perception information; the multimodal perception information includes: image, video, audio and posture information;

[0011] A fusion unit, used for preprocessing and fusing the multimodal sensing information to obtain fused information;

[0012] An identification unit, configured to identify the fusion information and obtain an identification result; the identification result includes at least one of the following: user surrounding environment information and user motion state information;

[0013] A generating unit is used to generate feedback information based on the recognition result.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] According to another aspect of the present disclosure, there is provided a wearable device, comprising:

[0021] A multimodal sensor group, used to collect multimodal perception information; the multimodal sensor group includes: a camera, a microphone and a posture sensor; the camera is used to collect images; the microphone is used to collect audio; the posture sensor is used to collect posture information;

[0022] A feedback device is used to provide feedback to the user based on the feedback information; wherein, the feedback information is generated by preprocessing and fusing the multimodal perception information, identifying the fused information after fusion, and based on the identified user surrounding environment information or user motion state information.

[0023] According to another aspect of the present disclosure, there is provided an intelligent blind guiding system, comprising: an intelligent blind guiding device according to any one of the embodiments of the present disclosure, and a wearable device according to any one of the embodiments of the present disclosure.

[0024] The disclosed embodiments can provide users with more comprehensive, accurate, and reliable perception and safe navigation services, ensuring the safety of users' travel, thereby improving the independence and quality of life of users.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0027] Figure 1 Schematic diagram of an application scenario of the intelligent blind guiding method provided by an embodiment of the present disclosure;

[0028] Figure 2 It is a schematic diagram of the implementation process of the intelligent blind guiding method provided by the embodiment of the present disclosure;

[0029] Figure 3 is a schematic diagram of an implementation process of obtaining posture information according to an embodiment of the present disclosure;

[0030] Figure 4 is a schematic diagram of an implementation flow of generating voice feedback information according to an embodiment of the present disclosure;

[0031] Figure 5 is a schematic diagram of an implementation flow of generating vibration feedback information according to an embodiment of the present disclosure;

[0032] Figure 6 is a schematic diagram of an implementation flow of generating pressure feedback information according to an embodiment of the present disclosure;

[0033] Figure 7 is a schematic diagram of an intelligent blind guiding device according to an embodiment of the present disclosure;

[0034] Figure 8 is a block diagram of an electronic device for implementing the intelligent blind guiding method of the embodiment of the present disclosure;

[0035] Fig. 9 is a schematic diagram of a wearable device according to an embodiment of the present disclosure;

[0036] Fig.10 is a schematic diagram of an intelligent blind guiding system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] With the development of science and technology, guide devices in related technologies can provide visual assistance to the visually impaired to assist them in daily activities.

[0039] However, the accuracy and reliability of the blind guide devices in the related art are low in the assistance process.

[0040] In order to overcome at least one of the above problems, the embodiments of the present disclosure provide an intelligent blind guiding method, device and equipment, which obtains multimodal perception information, preprocesses and fuses the multimodal perception information, identifies the fused information, and generates feedback information based on the identified information, thereby providing more comprehensive, accurate and reliable perception and safe navigation services for users such as the visually impaired, which is conducive to enhancing the independence and quality of life of the visually impaired.

[0041] Figure 1 This is a schematic diagram of an application scenario of the intelligent blind-guiding method provided in the embodiment of the present disclosure. Figure 1 , the scenario includes: a wearable device and a server, etc. The wearable device and the server are connected by wireless communication. The wearable device is used to be worn by the user and to obtain multimodal perception information such as images, videos, audios and posture information of the wearable device around the user. The server can obtain the multimodal perception information collected by the wearable device, preprocess and fuse the multimodal perception information to obtain fusion information; identify the fusion information to obtain the user's surrounding environment information and / or the user's motion state information; based on the user's surrounding environment information and / or the user's motion state information, generate feedback information and send it to the wearable device. The wearable device provides feedback based on the received feedback information.

[0042] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. The embodiments of the present disclosure can be applied to any applicable scenario.

[0043] Figure 2 It is a schematic diagram of the implementation flow of the intelligent blind guiding method provided according to an embodiment of the present disclosure.

[0044] Please refer to Figure 2 The intelligent blind guiding method of the embodiment of the present disclosure includes:

[0045] S21: Acquire multimodal perception information; the multimodal perception information includes: image, video, audio and posture information;

[0046] S22: preprocessing and fusing the multimodal perception information to obtain fused information;

[0047] S23: Identify the fused information to obtain an identification result; the identification result includes at least one of the following: user surrounding environment information and user motion state information;

[0048] S24: Generate feedback information based on the recognition result.

[0049] The execution subject of this embodiment may be a server or a terminal, and the server or the terminal is connected to the wearable device in communication. The execution subject of this embodiment may also be a wearable device, or other devices with data processing capabilities and capable of communicating with the wearable device.

[0050] In step S21, multimodal sensing information can be obtained from a wearable device worn by the user. The wearable device includes at least one of the following forms: a head-mounted guide device for the blind, guide glasses, a guide bracelet, and a guide shoulder cover. A multimodal sensor group with sensing function is set in the wearable device.

[0051] In a specific implementation, the multimodal perception information can be obtained from one or more wearable devices worn by the user. For example, the multimodal perception information can be obtained from a head-mounted guide device for the blind and a guide bracelet worn by the user. For another example, the multimodal perception information can be obtained from a head-mounted guide device for the blind worn by the user.

[0052] Multimodal perception information may include: images. Images can be used to provide information about obstacles, traffic signs, road conditions, weather, buildings, etc. in the user's surrounding environment. Obstacles may include: poles, pedestrians, vehicles, and other objects that affect the user's walking.

[0053] The image may be acquired by a camera in the wearable device. If the acquired image may be acquired by at least two cameras in the wearable device, for example, by three cameras distributed along the circumference of the user's head, the images acquired by the at least two cameras may be fused. If the acquired image is acquired by a camera in the wearable device, the camera may be set in front of the wearable device, and the front refers to the direction in which the user is moving.

[0054] The multimodal sensing information may include: audio. The audio may include at least one of the following: vehicle driving sound, pedestrian footsteps, vehicle horn sound, alarm sound, etc. The audio may be used to provide information about emergency sounds in the user's surrounding environment, such as vehicle horn sound, alarm sound, etc. The audio may be collected by a microphone in the wearable device.

[0055] Multimodal perception information may include: video. The images in the video are used to provide visual information about the user's surroundings. The audio in the video can be used to provide sound information about the user's surroundings. The video may be collected by a module such as a microphone-integrated camera in a wearable device. The video may include: video of the user's surroundings, content played by multimedia devices around the user, etc.

[0056] The multimodal sensing information may include posture information. The posture information is used to confirm the posture of the wearable device to confirm the user's motion state; the user's motion state may include: walking, still, falling, running, turning, etc. The posture information may be collected by the posture sensor on the wearable device.

[0057] In a specific implementation, the multimodal perception information may include one or any combination of images, videos, audios, and posture information. In different places, the types of information in the multimodal perception information may be the same or different. For example, when a user is walking on the street, the multimodal perception information may include images, audios, and posture information. For another example, when a user is in a place such as a shopping mall or supermarket where multimedia equipment is installed, the multimodal perception information may include video and posture information. For another example, when a user is in a relatively quiet place, the multimodal perception information may include images and posture information.

[0058] In step S22, the multimodal perception information may be filtered and denoised to remove noise and interference signals. For example, a median filter may be used to remove salt and pepper noise in an image. For another example, a mean filter may be used to remove high-frequency noise in an audio.

[0059] The multimodal perception information can also be normalized, thereby converting data from at least two sources into a unified format and range to facilitate subsequent fusion.

[0060] The pre-processed multimodal perception information is fused, so that information from multiple dimensions can be comprehensively considered to improve the accuracy and reliability of perception. The algorithm used in the fusion process may include at least one of the following: Kalman filter, particle filter or deep learning model.

[0061] In step S23, a deep learning model based on CNN (Convolutional Neural Network) or RNN (Rerrent Neural Network) or LSTM (long short-term memory) can be used to analyze and identify the fused information to obtain the user's surrounding environment information and / or the user's motion status information.

[0062] For example, the fusion information can be input into a deep learning model based on CNN and RNN to obtain visual information such as obstacles and traffic signs on the road ahead of the user and sound information such as pedestrian walking sounds or vehicle driving sounds in the user's surrounding environment. The visual information and sound information constitute at least part of the user's surrounding environment information. Optionally, the user's surrounding environment information may include a semantic description of the user's surrounding environment to facilitate subsequent voice feedback.

[0063] For another example, by inputting the fused information into a deep learning model, the user's motion state information can be output, and the current user's motion state can be determined to be a fall.

[0064] In step S24, when it is determined based on the recognition result that feedback needs to be provided to the user, feedback information is generated. For example, when it is determined based on the user's surrounding environment information that guidance is needed for the user, corresponding feedback information is generated to guide the user. For another example, when it is determined based on the user's motion state information that the user's motion state is a fall, corresponding feedback information is generated to prompt the user or pedestrians around the user.

[0065] The feedback information may include at least one of the following: voice feedback information, vibration feedback information, and pressure feedback information. Voice feedback information is used to trigger the wearable device to provide voice feedback to the user; for example, voice feedback information is used to trigger the wearable device to give a voice prompt to the user, such as "There is an obstacle ahead, please turn left". Vibration feedback information is used to trigger the wearable device to provide vibration feedback to the user; for example, vibration feedback information is used to trigger the vibration motor in the wearable device to output continuous slight vibrations to prompt the user that they need to turn. Pressure feedback information is used to trigger the wearable device to provide pressure feedback to the user; for example, when the road ahead of the user is narrow, the pressure feedback information is used to trigger the wearable device to provide greater pressure to the user, so that the user feels a heavier sense of pressure.

[0066] When the user is surrounded by noise, for example, when the decibel of the audio around the user is higher than the set value, the feedback information may include voice feedback information, and at least one of vibration feedback information and pressure feedback information, so as to ensure that the user can accurately receive feedback. When the user is surrounded by quietness, the feedback information may include voice feedback information. In other scenarios, the feedback information may include vibration feedback information or pressure feedback information according to the needs and habits of the user.

[0067] In specific implementation, if the recognition result includes the user's surrounding environment information, voice feedback information can be generated based on the semantic description of the user's surrounding environment information; if it is determined that the decibel of the audio around the user is higher than the set value at this time, vibration feedback information can also be generated. If the recognition result includes the user's motion state information, it can be determined based on the user's motion state information whether the user's motion state is a motion state that is not conducive to the user's safety, such as falling or collision; if so, voice feedback information can be generated, so that the wearable device can give voice prompts to the user or pedestrians around him based on the voice feedback information.

[0068] The intelligent blind guiding method provided by the embodiment of the present disclosure obtains multimodal perception information; the multimodal perception information includes: images, videos, audio and posture information; pre-processes and fuses the multimodal perception information to obtain fused information; recognizes the fused information to obtain a recognition result; the recognition result includes at least one of the following: user's surrounding environment information, user's motion state information; generates feedback information based on the recognition result, so as to provide users with more comprehensive, accurate and reliable perception and safe navigation services, ensure the user's travel safety, and thus improve the user's independence and quality of life.

[0069] Please refer to Figure 3 In some embodiments, step S21, obtaining posture information may include:

[0070] S31: Acquire acceleration information, angular velocity information and magnetic field direction information;

[0071] S32: Correct the acceleration information and angular velocity information based on the magnetic field direction information to obtain posture information.

[0072] In step S31, the acceleration information may be collected by an accelerometer in the wearable device worn by the user, and the acceleration information may include accelerations of the three axes X, Y, and Z. The X axis corresponds to the front-to-back direction of the user, the Y axis corresponds to the left-to-right direction of the user, and the Z axis corresponds to the up-to-down direction of the user. The acceleration information is used to determine the motion state and direction of the wearable device. When the device is stationary, the acceleration information can also be used to determine the tilt angle of the wearable device. For example, when the acceleration of the Z axis is close to -9.8 m / s 2 (m / s 2 When φ(m / s2) is measured, it can be determined that the wearable device is in a flat state.

[0073] The angular velocity information may be collected by a gyroscope in a wearable device worn by a user, and the angular velocity information may include the angular velocity of the three axes X, Y, and Z. The angular velocity information may be used to determine the rotation angle and rotation direction of the wearable device. The magnetic field direction information may be collected by a magnetometer in a wearable device worn by a user, and the magnetic field direction information is used to determine the direction of the geomagnetic field.

[0074] In step S32, the acceleration information and angular velocity information are corrected based on the geomagnetic field direction in the magnetic field direction information, and the corrected acceleration information and angular velocity information are used as posture information, so that the posture of the wearable device can be determined according to the posture information, and then the user's motion state can be determined.

[0075] In this embodiment, based on the acceleration information and angular velocity information, the three-dimensional posture information of the wearable device can be more accurately determined, such as the tilt angle, pitch angle, roll angle, etc. Moreover, by correcting the acceleration information and angular velocity information using the magnetic field direction information, the error caused by the offset of the accelerometer or gyroscope can be avoided, thereby improving the accuracy of the posture information.

[0076] In some embodiments, in order to further improve the comprehensiveness and accuracy of the multimodal sensing information, the multimodal sensing information may also include: ultrasonic detection information and / or infrared detection information.

[0077] The ultrasonic detection information can be used to determine the distance between the obstacle and the user, and the position of the obstacle relative to the user. The ultrasonic detection information can be collected by an ultrasonic sensor in a wearable device worn by the user. The ultrasonic sensor can be installed in front and on both sides of the wearable device.

[0078] The infrared detection information is used to detect the thermal radiation of surrounding objects, and then used to confirm pedestrians in the user's surrounding environment, pedestrian density, and the distance between pedestrians and the user, etc. The infrared detection information can be collected by an infrared sensor in a wearable device worn by the user.

[0079] In this embodiment, by fusing ultrasonic detection information with information such as images, the user can be accurately guided to avoid obstacles; by fusing infrared detection information with information such as images and audio, the situation of pedestrians around the user can be accurately determined.

[0080] In some embodiments, in order to further improve user safety, the intelligent blind guiding method also includes: obtaining the user's physical health status information; the physical health status information includes at least one of the following: body temperature, heart rate, blood oxygen saturation; when it is determined that the user is in an abnormal state based on the user's motion status information or physical health status information, generating a warning message or a help message; the warning message is used to prompt the user and pedestrians around him; the help message is used to be sent to the terminal of a preset contact.

[0081] This embodiment can obtain the user's physical health status information from the wearable device worn by the user. For example, the user's heart rate can be obtained from the heart rate sensor in the wearable device; the user's body temperature can be obtained from the temperature sensor in the wearable device; the user's blood oxygen saturation can be obtained from the blood oxygen saturation sensor of the wearable device. Of course, the user's physical health status information is not limited to this, and can be set according to actual needs.

[0082] When the user is determined to be in an abnormal state such as falling, collision or high fever according to the obtained user motion state information or physical health state information, a warning message can be generated, and the warning message is used to trigger the wearable device to remind the user or pedestrians around him. In particular, when the user is determined to be in a coma or has a fast heartbeat according to the user motion state information or physical state information, the warning message can trigger the wearable device to remind with a preset decibel sound. In particular, the more dangerous the user's current situation is, the higher the preset decibel.

[0083] In this embodiment, while generating the warning message, a help message for sending to the terminal of the preset contact may also be generated. Alternatively, after the warning message is generated, if the user does not resume standing or walking within a preset time period, or if the user does not receive an instruction to cancel the help request, a help message for sending to the terminal of the preset contact is generated.

[0084] This embodiment can provide more comprehensive and reliable auxiliary services for users' travel by monitoring the user's physical health status and setting up corresponding warning and assistance mechanisms, which helps to more comprehensively protect the user's travel safety.

[0085] In some embodiments, step S22, preprocessing the multimodal perception information, may include at least one of the following:

[0086] Step a1: filtering and / or normalizing the posture information;

[0087] Step a2: denoising and / or segmenting the image or video;

[0088] Step a3: filter the audio for noise;

[0089] Step a4: Synchronize the image, video, audio and pose information based on the timestamp information.

[0090] In step a1, this embodiment can remove unnecessary noise or unstable signals by filtering the posture information.

[0091] Optionally, when the posture information includes information from multiple sources, for example, when the posture information includes acceleration information from an accelerometer and angular velocity information from a gyroscope, the acceleration information and angular velocity information may also be normalized to ensure consistency in data format and range.

[0092] In step a2, taking an image as an example, in the denoising process, a median filter may be used to remove salt and pepper noise in the image; or a maximum filter may be used to process pepper noise, and a minimum filter may be used to process salt noise. This embodiment can improve image quality and the accuracy of subsequent image recognition by performing denoising on the image.

[0093] In the segmentation process, an edge detection-based segmentation method or a deep learning-based image segmentation model can be used to segment the image. This embodiment separates objects located in different areas and having different characteristics by segmenting the image, which is conducive to improving the accuracy of subsequent image recognition results.

[0094] The implementation method of preprocessing the pictures in the video may be the same as or similar to the above example.

[0095] In step a3, a filter may be used to remove noise in the audio. For example, a mean filter may be used to remove high-frequency noise in the audio. Alternatively, a fast Fourier transform (FFT) may be used to denoise the audio.

[0096] This embodiment can remove unnecessary noise or interference from the audio signal by performing noise filtering on the audio, so as to improve the quality and clarity of the audio. For example, by removing external noise from the audio through noise filtering, it is beneficial to improve the clarity of the emergency sounds of concern, such as sirens and alarms.

[0097] In step a4, a unified timestamp is used to synchronize the various information in the multimodal perception information, ensuring that the information from multiple sources can be aligned in the same time and space, thereby facilitating the improvement of the accuracy of the subsequent fusion information.

[0098] After preprocessing the image, video, audio, posture information, etc., features from at least two perception information can be extracted respectively, and then the extracted features can be fused using a fusion algorithm to integrate the image, video, audio, posture information, etc. into a unified perception model.

[0099] In some embodiments, step S22, fusing the multimodal perception information, may include:

[0100] Step b1: extract features from the image to obtain image features;

[0101] Step b2: extracting features from the audio to obtain audio features;

[0102] Step b3: extracting features from the video to obtain image features and / or audio features;

[0103] Step b4: extracting features from the posture information to obtain posture features;

[0104] Step b5: Fuse the image features, audio features and posture features to obtain fusion information.

[0105] In step b1, taking an image as an example, a deep learning model can be used to extract features of a target object from the image, such as features of obstacles, features of traffic signs, etc. Alternatively, a histogram of directional gradients can be used to extract features of pedestrians from the image. When processing a video, the implementation method can be the same as the above example.

[0106] In step b2, key audio features may be extracted from the audio using short-time Fourier transform or Mel-frequency cepstral coefficients.

[0107] In step b3, the process of extracting features from the video may be the same or similar to the aforementioned steps b1 and b2, and will not be described in detail herein.

[0108] In step b4, generally speaking, the posture information may include information from multiple measuring devices. Taking the posture information including acceleration information and angular velocity information as an example, when extracting features, the timing features and acceleration may be extracted from the acceleration information, and the timing features and three-dimensional posture features such as angular velocity may be extracted from the angular velocity information; or, the acceleration information and angular velocity information may be synchronized based on time, and then the timing features, acceleration and three-dimensional posture features may be extracted from the synchronized information.

[0109] In step b5, a fusion algorithm such as a weighted average method, a Kalman filter, a multi-Bayesian estimation method, or a particle filter method may be used to fuse the extracted at least two features. For scenes that require analysis of more complex features, a deep learning model may also be used to integrate image features, three-dimensional posture features, and time series features, thereby obtaining a higher quality fusion result, which is convenient for improving the accuracy of subsequent image recognition results or speech recognition results.

[0110] In this embodiment, by extracting and fusing features of multimodal perception information, more comprehensive, accurate and reliable perception information can be provided for subsequent identification, which is conducive to reducing misjudgments and missed judgments in the subsequent identification process, thereby improving the accuracy of identification.

[0111] After the multimodal perception information is fused to obtain fused information, the fused information can be analyzed and identified using a recognition model based on deep learning to obtain the user's surrounding environment information and / or user motion status information, providing an accurate and reliable reference basis for the subsequent generation of feedback information.

[0112] In some embodiments, the user's surrounding environment information includes at least one of the following: visual information of the user's surrounding environment and audio information of the user's surrounding environment. Exemplarily, step S23, identifying the fused information to obtain a recognition result, includes: inputting the fused information into a recognition model based on CNN and RNN for recognition, so as to obtain visual information of the user's surrounding environment and / or audio information of the user's surrounding environment.

[0113] Among them, the visual information of the user's surrounding environment can assist the user in confirming obstacles, road conditions, traffic signs, buildings, etc. in the surrounding environment; the visual information of the user's surrounding environment can assist the user in confirming the footsteps of pedestrians, the sound of vehicles driving, the sound of vehicle horns, the sound of alarms, etc. in the surrounding environment.

[0114] This embodiment adopts a recognition model based on a CNN and RNN fusion architecture, which can learn the common patterns of multiple types of perceptual information, and can help to identify the common association of static information (such as images) and dynamic information (such as motion), so that the recognition model can flexibly process more scenes in time and space dimensions. Moreover, this embodiment can provide users with comprehensive, accurate and reliable perception by identifying visual information and audio information of the user's surrounding environment, thereby helping to improve the user's travel safety.

[0115] In order to improve the accuracy of the recognition results of the above recognition model, in the process of training the recognition model, images, audio and posture information in scenes associated with the user's daily life can be used as training samples, and a loss function can be introduced to optimize the model parameters. In some embodiments, the intelligent blind guidance method can also include:

[0116] Step c1: obtaining a training data set for the recognition model; the training data set includes: an image and / or video and / or audio of at least one target location, and position information corresponding to at least one target motion state;

[0117] Step c2: input the training data set into the recognition model for training and obtain the output result;

[0118] Step c3: Based on the output results, optimize the model parameters of the recognition model.

[0119] In step c1, the target place includes at least one of the following: accommodation and social places, beauty places, cultural and entertainment places, sports and recreation places, shopping places, cultural exchange places, medical and transportation places, etc. Among them, accommodation and social places include: restaurants, hotels, cafes, tea houses, etc. Beauty places include: barber shops and beauty salons, etc. Cultural and entertainment places include: theaters, concert halls, etc. Sports and recreation places include: gymnasiums, swimming pools, parks, etc. Shopping places include: shopping malls, supermarkets, bookstores, etc. Cultural exchange places include: exhibition halls, museums, libraries, etc. Medical and transportation places include: hospitals, streets, waiting rooms (planes, ships), public transportation vehicles (cars, trains, airplanes and ships), public transportation stations, etc.

[0120] This embodiment can collect images and / or videos and / or audio of the target place under multiple weather conditions, and the weather conditions may include sunny days, cloudy days, rainy and snowy days, etc.; it can collect images and / or videos and / or audio of the target place under multiple lighting conditions, such as backlight conditions, and the lighting conditions may include multiple light intensities and backlight; it can collect images and / or videos and / or audio of the target place in multiple time periods, and the time periods may include time periods corresponding to morning, noon, evening, night, etc.

[0121] Optionally, this embodiment may also utilize data enhancement technology to process the training data set, such as randomly adding noise to the images in the training data set, or adding non-ideal conditions such as occlusion and blur, to enhance the performance of the model in non-ideal scenarios.

[0122] This embodiment can also collect posture information corresponding to at least one type of action in the user's daily life activities as posture information corresponding to the target motion state. The target motion state may include at least one of the following: falling, collision, walking, running, standing still, turning, etc.; wherein, falling may include at least one of the following: falling forward, falling backward, falling sideways, and falling from a step or platform.

[0123] In one possible implementation manner, this embodiment may also implement negative sample enhancement by simulating an interference scene or simulating an abnormal motion state of a user.

[0124] In some examples, at least one interference scenario can be simulated, for example, physical collisions in multiple directions, complex road conditions, interference noise, and blurred vision; wherein the interference noise includes at least one of the following: background noise, microphone error, and optical noise; blurred vision can include at least one of the following: blurred vision on rainy days, blurred vision under dim lights, and blurred vision in high-speed motion scenes. Therefore, the training data set also includes: images and / or videos and / or audios of simulated interference scenarios, thereby providing more complex samples for the recognition model and improving the robustness and anti-interference ability of the recognition model.

[0125] In other examples, abnormal motion states such as user falls, turns, collisions, etc. can be simulated to provide more difficult samples for the recognition model, so that the recognition model can capture abnormal behaviors or predict emergencies that users may encounter. Therefore, the training data set also includes: the pose information of simulated abnormal motion states, thereby enhancing the recognition model's ability to recognize and predict rare but important or urgent scenes, and facilitating timely feedback on such scenes.

[0126] This embodiment obtains a richer training data set by collecting or simulating images, videos, audio and posture information in a variety of scenes, so that the recognition model can learn richer environmental patterns and improve the generalization ability of the recognition model in complex scenes.

[0127] In step c2, the acquired training data set is input into the established recognition model for prediction, and the output prediction result can be used to optimize the model parameters in the recognition model.

[0128] In step c3, model parameters such as hyperparameters can be optimized based on the output results of the above steps. Exemplarily, a loss function can be determined based on the output results of the above steps and pre-set training labels; based on the loss function, the model parameters of the recognition model are adjusted until the total loss function is less than the set value. The loss function can include a cross entropy loss function or a mean square error loss function.

[0129] Optionally, in order to improve the imbalance between multimodal perception information and ensure timing consistency, this example can combine the weighted loss function, the regularization loss function and the KL (Kullback-Leibler divergence, also known as relative entropy) divergence loss function to improve the accuracy of the recognition results of the recognition model.

[0130] Exemplarily, step c3 may include:

[0131] Step c31: determining a total loss function based on the output result of the recognition model and the pre-set training label; the total loss function includes: a weighted loss function, a regularization loss function and a KL divergence loss function;

[0132] Step c32: Based on the total loss function, adjust the model parameters of the recognition model until the total loss function is less than the set value.

[0133] In step c31, during the fusion of multimodal perception information, the contribution of perception information from different sources to the final result may be different. Therefore, a dynamic weighting mechanism can be used to process the perception information of at least two modalities to enhance the adaptability of the recognition model in multimodal perception scenarios.

[0134] For example, in the initial stage, fixed weights are set for the image and the pose information, respectively, and the weights are used to reflect the importance of the corresponding information in the set scene. For example, the weight corresponding to the image and the weight corresponding to the pose information can be set equal.

[0135] During the training phase, the feedback from the recognition model during the training process is analyzed, and the weights corresponding to the image and the posture information are dynamically adjusted, so that the recognition model can automatically learn the contribution of multiple modal information to the final recognition result, ensuring that the weights can reflect the impact on the actual scene.

[0136] For example, in the scenario of identifying the user's motion state, the weight corresponding to the posture information can be greater than or equal to the weight corresponding to the image; in the scenario of identifying target objects such as obstacles, traffic signs, or pedestrians, the weight corresponding to the image can be greater than the weight of the posture information.

[0137] For example, when the quality of one modality perception information is low, the weight corresponding to the modality perception information can be dynamically reduced. In specific implementation, when the image is too dark, the weight corresponding to the image can be reduced; when the noise in the posture information is greater than the set value, the weight of the posture information can be reduced.

[0138] After completing multiple forward calculations and back propagation processes, the recognition model can gradually and automatically optimize the weights corresponding to the information of multiple modalities, thereby reducing the negative impact on the recognition results caused by low information quality due to poor sensor performance.

[0139] For time series data such as acceleration information and angular velocity information in posture information, it is necessary to have the characteristic of smooth change in the time dimension. Therefore, this embodiment also introduces a regularized loss function in the total loss function.

[0140] For example, a time sliding window can be used to detect the differences in time series data between adjacent time frames, establish time consistency constraints between frames, and ensure that the change pattern between consecutive frames is a natural transition, such as a natural transition from walking to standing.

[0141] For situations where the change amplitude in adjacent time series frames is too large or the jump is abnormal, such as a fall is detected in one frame and the next frame detects the resumption of walking, penalties are imposed on the above situations to ensure the smoothness of the time series data.

[0142] In order to improve the difference between multimodal perception information, this embodiment also introduces the KL divergence loss function in the total loss function. The KL divergence loss function is used to measure the difference between the predicted distribution and the true distribution of at least two modal perception information, and reduce the difference between the two modal perception information by aligning the predicted distribution of at least two modal perception information, thereby maximizing the accuracy of the prediction result and enhancing the stability of the multimodal perception information collaboration. Among them, a penalty can be imposed when the distribution difference of the predicted data is greater than the set value.

[0143] Optionally, the sum of the weighted loss function, the regularized loss function and the KL divergence loss function can be used as the total loss function. In addition, weights can be assigned to the weighted loss function, the regularized loss function and the KL divergence loss function respectively according to actual needs.

[0144] By introducing a weighted loss function, this embodiment can improve the robustness of multimodal perception information fusion and enable the recognition model to make accurate contribution evaluation of the multimodal perception information; by introducing a regularized loss function, this embodiment can ensure coherent prediction in the time dimension and reduce the impact of sudden jitters and meaningless fluctuations on the recognition results of the recognition model; by introducing a KL divergence loss function, this embodiment can enhance the consistency of multimodal perception information and ensure the smoothness of the prediction results.

[0145] After obtaining the user's surrounding environment information by using the recognition model, a decision that meets the user's needs can be made based on the user's surrounding environment information to obtain feedback information for providing feedback to the user.

[0146] Please refer to Figure 4 In some embodiments, step S24: generating feedback information based on the user's surrounding environment information includes:

[0147] S41: performing path planning based on the user's surrounding environment information and the user's target location to obtain the user's target path;

[0148] S42: Generate voice feedback information based on the user's surrounding environment information and the target path.

[0149] In step S41, the user's current location can be determined based on the user's surrounding environment information. Obstacles around the user or dangerous areas on the road can also be determined based on the user's surrounding environment information, and then the area that the user needs to avoid can be determined. In the process of path planning based on the user's current location and target location, the above-mentioned areas need to be avoided.

[0150] In step S42, the voice feedback information may include at least one of the following: navigation information, category or location of obstacles, number or density or location of pedestrians, status or number or density or location of vehicles, traffic signs, etc.

[0151] For example, if the traffic light information is determined to be red or yellow based on the user's surrounding environment information, and pedestrians around have stopped moving, the voice feedback information can trigger the wearable device to announce to the user "There is an intersection ahead, the current traffic light is red (yellow), please stay and wait at the current location." If the traffic light information is determined to be green based on the user's surrounding environment information, the voice feedback information can trigger the wearable device to announce to the user "There is an intersection ahead, the current traffic light is green, please continue moving forward."

[0152] For another example, when it is determined that there is an obstacle in front of the user based on the user's surrounding environment information, the voice feedback information can trigger the wearable device to announce to the user "An obstacle is detected ahead, please stop (or go around to the left or right, or the route is being replanned for you)".

[0153] In this embodiment, a route is planned for the user based on the user's surrounding environment information, and through voice feedback, the user is more intuitively assisted to accurately avoid areas with obstacles or dangerous areas, etc., thereby guiding the user to reach the destination safely.

[0154] In order to further enhance the user experience, real-time interaction with the user can also be performed based on the user's surrounding environment information. Exemplarily, the intelligent blind guiding method can also include: receiving the user's voice command; performing semantic recognition on the voice command; and generating corresponding voice feedback information according to the semantic recognition result.

[0155] Among them, when the semantic recognition result is to adjust the target position, the target path is adjusted according to the target position, and voice feedback information is generated based on the adjusted target path; when the semantic recognition result is a query request, voice feedback information is generated based on the query request and the user's surrounding environment information.

[0156] In an application scenario of this embodiment, when a user wears a wearable device for traveling, the user's voice command can be obtained through the wearable device, and the voice command is used to request navigation information query or to request repeated broadcasting of navigation information; voice feedback information is generated based on the corresponding navigation information, and the voice feedback information can trigger the wearable device to broadcast navigation information to the user. For example, when the semantic recognition result of the received voice command is to adjust the target location, the path can be replanned and voice feedback information can be generated. The voice feedback information can trigger the wearable device to broadcast to the user "Your destination is XXX shopping mall, 3.3 kilometers remaining, and it is expected to take 50 minutes."

[0157] In another application scenario, when a user wears a wearable device while traveling, the wearable device can obtain the user's voice command, which is used to request a semantic description of the current surrounding environment; voice feedback information is generated based on the semantic description of the surrounding environment, and the voice feedback information can trigger the wearable device to broadcast a description of the user's surrounding environment to the user. For example, when the semantic recognition result of the received voice command is which shops are near the current location, the generated voice feedback information can trigger the wearable device to broadcast to the user "XXX convenience store, XXX restaurant" near your current location.

[0158] In other application scenarios, when a user is traveling while wearing a wearable device, based on the multimodal perception information obtained through the wearable device, it is determined that the current user motion state is a collision or a fall, or the user motion state is predicted to be a collision or a fall, and voice feedback information is generated based on the current user motion state or the predicted user motion state. The voice feedback information can trigger the wearable device to broadcast prompts to the user and pedestrians around him.

[0159] In scenarios where the user's surrounding environment is noisy and affects the user's hearing, in order to ensure that the user receives feedback in a timely and accurate manner, vibration feedback or pressure feedback can also be provided to the user.

[0160] Please refer to Figure 5 In some embodiments, step S24: generating feedback information based on the user's surrounding environment information includes:

[0161] S51: Determine the urgency of the content to be fed back based on the user's surrounding environment information and the target path;

[0162] S52: Determine the vibration intensity based on the urgency level; the vibration intensity is positively correlated with the urgency level;

[0163] S53: Generate vibration feedback information based on the vibration intensity.

[0164] In step S51, when it is determined that there are obstacles or waterlogged areas within a preset range in front of the user that may affect the user's safety based on the user's surrounding environment information and the direction of travel, navigation information to be fed back can be generated based on the existence of obstacles or waterlogged areas within the preset range in front of the user, and the degree of urgency can be determined based on the distance between the obstacle or waterlogged area and the user; wherein, the closer the distance between the obstacle or waterlogged area and the user, the higher the degree of urgency.

[0165] In step S52, the higher the urgency level, the greater the vibration intensity.

[0166] In step S53, in some examples, vibration feedback information is generated based on the vibration intensity. The vibration feedback information can trigger the vibration motor in the wearable device to provide a corresponding vibration intensity, thereby conveying the urgency of the situation to the user. In specific implementation, the user can obtain timely and accurate feedback by combining the voice feedback information and the vibration feedback information.

[0167] In other examples, multiple vibration modes can be set, and different vibration modes correspond to different feedback contents. For example, the vibration mode can include intermittent vibration and continuous vibration; intermittent vibration is used to prompt the user to avoid obstacles, and continuous vibration is used to prompt the user to turn. In this way, vibration feedback information can be generated according to the content to be fed back and the vibration intensity.

[0168] In addition, in order to meet the personalized needs of users, it is also possible to receive a user's request to set the vibration mode and vibration intensity, and receive a user's request to set the feedback content corresponding to the vibration mode and vibration intensity.

[0169] In this embodiment, vibration feedback can be combined with voice feedback to ensure that users receive feedback information in a timely and accurate manner, and a variety of feedback methods can be provided to users to meet the needs and habits of different users and improve user experience.

[0170] Please refer to Figure 6 In some embodiments, step S24: generating feedback information based on the user's surrounding environment information includes:

[0171] S61: Determine a passable space in front of the user based on the user's surrounding environment information and the target path;

[0172] S62: Determine the pressure acting on the user based on the passable space; the pressure is positively correlated with the passable space;

[0173] S63: Generate pressure feedback information based on the pressure.

[0174] In step S61, when it is determined based on the user's surroundings and the direction of travel that the road ahead of the user is narrow or crowded, it can be determined that the passable space ahead of the user is small. If it is determined that the road ahead of the user is wide and open, it can be determined that the passable space ahead of the user is appropriate.

[0175] In step S62, when the passable space in front of the user is smaller, the pressure exerted by the wearable device on the user can be greater, thereby improving the user's immersion. When the passable space in front of the user is appropriate, the pressure exerted by the wearable device on the user can be small or zero.

[0176] In step S63, the pressure feedback information can trigger the pressure applying device in the wearable device to provide corresponding pressure. For example, the narrower the road in front of the user, or the more crowded the pedestrians on the road in front of the user, the greater the pressure applied to the user by the wearable device. The wearable device may include a wristband.

[0177] In addition, in order to meet the personalized needs of users, it is also possible to receive a user's request to set the pressure level, and receive a user's request to set the feedback content corresponding to different pressure levels.

[0178] In this embodiment, pressure feedback can be combined with voice feedback to ensure that users receive feedback information in a timely and accurate manner, and a variety of feedback methods can be provided to users to meet the needs and habits of different users and improve user experience.

[0179] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0180] Please refer to Figure 7 The embodiment of the present disclosure further provides an intelligent blind guiding device 70, comprising:

[0181] An acquisition unit 71 is used to acquire multimodal perception information; the multimodal perception information includes: image, video, audio and posture information;

[0182] A fusion unit 72, used for preprocessing and fusing the multimodal sensing information to obtain fused information;

[0183] The recognition unit 73 is used to recognize the fusion information and obtain a recognition result; the recognition result includes at least one of the following: user surrounding environment information and user motion state information;

[0184] The generating unit 74 is used to generate feedback information based on the recognition result.

[0185] In some embodiments, the fusion unit is configured to implement at least one of the following:

[0186] Filtering and / or normalizing the pose information;

[0187] Denoise and / or segment images or videos;

[0188] Filter noise from audio or video;

[0189] Synchronize image, video, audio and pose information based on timestamp information.

[0190] In some embodiments, the fusion unit is used to:

[0191] Extract features from the image to obtain image features;

[0192] Extract features from the audio to obtain audio features;

[0193] Extracting features from the video to obtain image features and / or audio features;

[0194] Extracting features from the posture information to obtain posture features;

[0195] The image features, audio features and posture features are fused to obtain fused information.

[0196] In some embodiments, the acquisition unit is used to:

[0197] Obtain acceleration information, angular velocity information and magnetic field direction information;

[0198] The acceleration information and angular velocity information are corrected based on the magnetic field direction information to obtain the posture information.

[0199] In some embodiments, the identification unit is used to:

[0200] The fused information is input into a recognition model based on a convolutional neural network and a recurrent neural network for recognition, thereby obtaining visual information of the user's surrounding environment and audio information of the user's surrounding environment.

[0201] In some embodiments, the intelligent blind guiding device further includes a first processing unit, which is used to:

[0202] Acquire a training data set for the recognition model; the training data set includes: an image and / or video and / or audio of at least one target location, and position information corresponding to at least one target motion state;

[0203] Input the training data set into the recognition model for training and obtain the output result;

[0204] Based on the output results, the model parameters of the recognition model are optimized.

[0205] In some embodiments, the training data set also includes: images and / or videos and / or audios of simulated interference scenes; and / or posture information of simulated abnormal motion states.

[0206] In some embodiments, the first processing unit is specifically configured to:

[0207] Based on the output results and the pre-set training labels, determine the total loss function; the total loss function includes: weighted loss function, regularization loss function and KL divergence loss function;

[0208] Based on the total loss function, adjust the model parameters of the recognition model until the total loss function is less than the set value.

[0209] In some embodiments, the generating unit is specifically configured to:

[0210] Perform path planning based on the user's surrounding environment information and the user's target location to obtain the user's target path;

[0211] Generate voice feedback information based on the user's surrounding environment information and target path.

[0212] In some embodiments, the generating unit is specifically configured to:

[0213] Determine the urgency of obstacles or dangerous areas around the user based on the user's surrounding environment information and target path;

[0214] Determine the vibration intensity based on the urgency level; the vibration intensity is positively correlated with the urgency level;

[0215] Vibration feedback information is generated based on the vibration intensity.

[0216] In some embodiments, the generating unit is specifically configured to:

[0217] Determine the passable space in front of the user based on the user's surrounding environment information and target path;

[0218] Determine the pressure acting on the user based on the passable space; pressure is positively correlated with the passable space;

[0219] Pressure feedback information is generated based on the pressure.

[0220] In some embodiments, the multimodal sensing information also includes: ultrasonic detection information and / or infrared detection information.

[0221] In some embodiments, the intelligent blind guiding device further includes a second processing unit, which is used to:

[0222] Obtain the user's physical health status information; the physical health status information includes at least one of the following: body temperature, heart rate, blood oxygen saturation

[0223] When it is determined that the user is in an abnormal state based on the user's motion status information or physical health status information, a warning message or a help message is generated; the warning message is used to prompt the user and pedestrians around him; the help message is used to be sent to the terminal of the preset contact.

[0224] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0225] The present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0226] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0227] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0228] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0229] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the intelligent blind guiding method. For example, in some embodiments, the intelligent blind guiding method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the intelligent blind guiding method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the intelligent blind guiding method in any other appropriate manner (e.g., by means of firmware).

[0230] Please refer to Fig. 9 The present disclosure further provides a wearable device 900, comprising:

[0231] A multimodal sensor group 910 is used to collect multimodal perception information; the multimodal sensor group 910 includes: a camera 911, a microphone 912 and a posture sensor 913;

[0232] The feedback device 920 is used to provide feedback to the user according to the feedback information; wherein, the multimodal perception information is preprocessed and fused, the fused information is identified, and the feedback information is generated based on the identified user surrounding environment information or user motion state information.

[0233] Among them, the camera 911 can use a high-definition camera to capture clearer images. Optionally, the resolution of the image captured by the camera 911 is at least 12 million pixels. Generally speaking, the viewing angle range of the camera 911 is about 120° so that the road, obstacles, traffic signals and other information in front of the user can be clearly captured. The camera 911 can be installed in front of the wearable device; or, the camera can have multiple cameras, and the multiple cameras 911 can be distributed along the circumference of the wearable device to be able to capture images within a 360° range, wherein the circumference of the wearable device is the same as the circumference of the user's head or body.

[0234] The sensitivity of the microphone 912 may be 40 decibels, and the frequency response range may be 16-20000 Hz, so that the sound around the user can be captured more clearly. Optionally, the sensitivity of the microphone 912 may be 40 decibels, and the frequency response range may be 20-20000 Hz. The camera 911 and the microphone 912 may be set independently or integrally.

[0235] In some embodiments, the posture sensor also includes: an accelerometer, a gyroscope and a magnetometer; the accelerometer is used to collect acceleration information of three axes; the gyroscope is used to collect angular velocity information of three axes; and the magnetometer is used to obtain magnetic field direction information.

[0236] The posture sensor 913 may include an accelerometer, and accordingly, the posture information may include acceleration information. Generally speaking, the acceleration generated by pedestrians during normal walking is usually within ±2g. g represents gravity acceleration, 1g = 9.8m / s 2 , m / s 2 represents meters per second squared). When a pedestrian falls or experiences an accident, the acceleration usually increases instantly. Therefore, this embodiment can more accurately determine whether the pedestrian is in a state of falling or the like through acceleration information.

[0237] In some examples, in order to take into account the accuracy of the perception results, low equipment cost and simple equipment structure, the measurement range of the accelerometer can be ±8g with an accuracy of 0.1mg. Among them, the measurement range of ±8g can better cover most fall situations, such as falling forward, falling backward or slipping sideways.

[0238] In other examples, in order to further improve the accuracy of the perception results, the measurement range of the accelerometer can be ±16g with an accuracy of 0.1mg, so as to accurately determine the user's motion state. Among them, the measurement range of ±16g can cover a more comprehensive fall situation, for example, it can cover more extreme and violent fall situations.

[0239] In some embodiments, the multimodal sensor group 910 further includes an ultrasonic sensor 914 and an infrared sensor 915 .

[0240] Among them, the detection range of the ultrasonic sensor can be 5 meters and the accuracy can be 1 centimeter, so it can accurately detect close obstacles such as telephone poles, road signs, trash cans, etc.

[0241] The infrared sensor has a detection distance of 3 meters and a temperature detection accuracy of 0.5 degrees Celsius, which can accurately distinguish the thermal radiation of the human body and other objects.

[0242] In some embodiments, the feedback device 920 includes:

[0243] The speaker 921 is used to provide voice feedback to the user when the feedback information includes voice feedback information;

[0244] a vibration motor 922, for providing vibration feedback to the user when the feedback information includes vibration feedback information;

[0245] The pressure control device 923 is used to provide pressure feedback to the user when the feedback information includes pressure feedback information.

[0246] In some embodiments, the wearable device includes at least one of the following: a head-mounted guide device for the blind, guide glasses, a guide bracelet, and a guide shoulder bag.

[0247] In the wearable device of the embodiment of the present disclosure, the same points as those in the aforementioned embodiment will not be repeated here.

[0248] Please refer to Fig.10 The present disclosure also provides an intelligent blind guide system 1000, including: an intelligent blind guide device 1001 as in any of the aforementioned embodiments, and a wearable device 1002 as in any of the aforementioned embodiments.

[0249] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0250] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0251] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0252] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0253] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0254] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0255] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0256] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An intelligent blind-guiding method, comprising: Acquire multimodal perception information; The multimodal perception information includes: image, video, audio and posture information; Preprocessing and fusing the multimodal sensing information to obtain fused information; Identify the fused information to obtain an identification result; the identification result includes at least one of the following: user surrounding environment information and user motion state information; Feedback information is generated based on the recognition result.

2. The method according to claim 1, wherein: The multimodal sensing information also includes: ultrasonic detection information and / or infrared detection information.

3. The method according to claim 1, wherein: Get pose information, including: Obtain acceleration information, angular velocity information and magnetic field direction information; The acceleration information and angular velocity information are corrected based on the magnetic field direction information to obtain posture information.

4. The method according to claim 1, wherein: The identifying the fusion information to obtain an identification result includes: The fused information is input into a recognition model based on a convolutional neural network and a recurrent neural network for recognition, thereby obtaining visual information of the user's surrounding environment and audio information of the user's surrounding environment.

5. The method according to claim 4, wherein: Also includes: Obtaining a training data set for the recognition model; The training data set includes: images, videos, and audios of at least one target location, and position information corresponding to at least one target motion state; Inputting the training data set into the recognition model for training to obtain an output result; Based on the output result, the model parameters of the recognition model are optimized.

6. The method according to claim 5, wherein: The training data set also includes: images and / or videos and / or audios of simulated interference scenes; and / or, posture information corresponding to the simulated abnormal motion state.

7. The method according to claim 5, wherein: The adjusting the model parameters of the recognition model based on the output result includes: Based on the output result and the pre-set training label, a total loss function is determined; the total loss function includes: a weighted loss function, a regularization loss function and a KL divergence loss function; Based on the total loss function, the model parameters of the recognition model are adjusted until the total loss function is less than a set value.

8. The method according to claim 1, wherein: Generating feedback information based on the user's surrounding environment information includes: Performing path planning based on the user's surrounding environment information and the user's target location to obtain the user's target path; Based on the user's surrounding environment information and the target path, voice feedback information is generated.

9. The method according to any one of claims 1 to 8, generating feedback information based on the user's surrounding environment information, comprising: Determining the urgency of the content to be fed back based on the user's surrounding environment information and the target path; determining a vibration intensity based on the urgency level; The vibration intensity is positively correlated with the degree of urgency; Vibration feedback information is generated based on the vibration intensity.

10. The method according to any one of claims 1 to 8, generating feedback information based on the user's surrounding environment information, comprising: Determine a passable space in front of the user based on the user's surrounding environment information and the target path; Based on the above, a pressure acting on the user can be determined through the space; The pressure is positively correlated with the passable space; Pressure feedback information is generated based on the pressure.

11. The method according to claim 1, wherein: Preprocessing the multimodal perception information includes at least one of the following: Filtering and / or normalizing the posture information; Performing noise reduction and / or segmentation on the image or video; performing noise filtering on the audio; Synchronize image, video, audio and pose information based on timestamp information.

12. The method according to claim 1, wherein: The multimodal perception information is integrated, including: Extracting features from the image to obtain image features; Extracting features from the audio to obtain audio features; Extracting features from the video to obtain image features and / or audio features; Performing feature extraction on the posture information to obtain posture features; The image features, audio features and posture features are fused to obtain fusion information.

13. The method according to claim 1, further comprising: Obtaining the user's physical health status information; the physical health status information includes at least one of the following: body temperature, heart rate, blood oxygen saturation; When it is determined that the user is in an abnormal state according to the user's exercise state information or physical health state information, generating a warning message or a help message; The warning information is used to remind the user and pedestrians around him; the help information is used to be sent to the terminal of the preset contact.

14. An intelligent blind-guiding device, comprising: An acquisition unit, used for acquiring multimodal perception information; The multimodal perception information includes: image, video, audio and posture information; A fusion unit, used for preprocessing and fusing the multimodal sensing information to obtain fused information; An identification unit, configured to identify the fusion information and obtain an identification result; the identification result includes at least one of the following: user surrounding environment information and user motion state information; A generating unit is used to generate feedback information based on the recognition result.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.

18. A wearable device comprising: A multimodal sensor group for collecting multimodal perception information; The multimodal sensor group includes: a camera, a microphone and a posture sensor; the camera is used to collect images; the microphone is used to collect audio; the posture sensor is used to collect posture information; A feedback device is used to provide feedback to the user based on the feedback information; wherein, the feedback information is generated by preprocessing and fusing the multimodal perception information, identifying the fused information after fusion, and based on the identified user surrounding environment information or user motion state information.

19. The wearable device according to claim 18, wherein: The posture sensor also includes: an accelerometer, a gyroscope and a magnetometer; the accelerometer is used to collect acceleration information; the gyroscope is used to collect angular velocity information; the magnetometer is used to obtain magnetic field direction information; Alternatively, the multimodal sensor group further includes: an ultrasonic sensor and an infrared sensor; the ultrasonic sensor is used to collect ultrasonic detection information; the infrared sensor is used to collect infrared detection information.

20. The wearable device according to claim 18, wherein: The feedback device comprises: A speaker, used for providing voice feedback to the user when the feedback information includes voice feedback information; a vibration motor, configured to provide vibration feedback to a user when the feedback information includes vibration feedback information; The pressure control device is used to provide pressure feedback to the user when the feedback information includes pressure feedback information.

21. The wearable device according to claim 18, wherein: The wearable device includes at least one of the following: a head-mounted guide device for the blind, guide glasses, a guide bracelet, and a guide shoulder bag.

22. An intelligent blind guide system, comprising: The intelligent blind guiding device as described in claim 14, and the wearable device as described in claims 18-21.

Images