Blind person navigation method and device based on visual touch cross-modal perception and medium

Abstract

The embodiment of the invention discloses a blind person navigation method and device based on visual touch cross-modal perception and a medium. The method is executed by a processor of the intelligent wearable device, the intelligent wearable device comprises intelligent glasses and an intelligent waistband, the intelligent waistband is provided with at least two vibration units, and the method comprises the following steps: acquiring a first environment image shot by a first camera based on the intelligent glasses, obtaining a second environment image shot by a second camera based on the intelligent glasses, and obtaining a target depth image according to the first environment image and the second environment image; the target depth image is divided into at least two area images, the minimum depth value of each area image is determined, each minimum depth value is converted into a voltage signal with the corresponding intensity, and the area images correspond to the vibration units; the voltage signals are transmitted to the intelligent waistband, the vibration units corresponding to the voltage signals are driven to vibrate, accurate conversion from vision to touch is achieved through multi-point vibration design, and cross-modal navigation experience is provided for a user.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of intelligent navigation technology, and in particular to a navigation method, device and medium for the blind based on visual-tactile cross-modal perception. Background Art

[0002] In related technologies, providing navigation prompts to the blind usually relies on voice prompts. However, this one-dimensional language expression becomes increasingly limited when faced with four-dimensional environmental information composed of three-dimensional space and time. Language, as a linear expression method, is difficult to fully and three-dimensionally depict the complex and ever-changing surrounding environment, especially in public places such as stations, supermarkets, and roads. These scenes are often full of rich spatial information and time dynamics. Existing blind navigation methods with voice prompts are difficult to ensure that the blind are provided with comprehensive, accurate and reliable environmental perception.

[0003] Therefore, a blind navigation method based on visual-tactile cross-modal perception is provided to achieve precise conversion from vision to touch through multi-point vibration design, providing users with a cross-modal navigation experience. Summary of the invention

[0004] The present invention provides a navigation method, device and medium for the blind based on visual-tactile cross-modal perception, which realizes precise conversion from vision to touch through multi-point vibration design, and provides users with a cross-modal navigation experience.

[0005] According to one aspect of the present invention, a blind navigation method based on visual-tactile cross-modal perception is provided, the method being executed by a processor of a smart wearable device, the smart wearable device comprising smart glasses and a smart belt, the smart belt being provided with at least two vibration units, the method comprising:

[0006] Acquire a first environment image taken by a first camera of the smart glasses, acquire a second environment image taken by a second camera of the smart glasses, and obtain a target depth image based on the first environment image and the second environment image;

[0007] Dividing the target depth image into at least two regional images, determining a minimum depth value of each of the regional images, and converting each of the minimum depth values ​​into a voltage signal of a corresponding strength based on a preset corresponding relationship between the depth value size and the voltage signal strength, wherein the regional image corresponds to the vibration unit;

[0008] Each of the voltage signals is transmitted to the smart belt, and a vibration unit corresponding to the voltage signal is driven to vibrate at the intensity of the voltage signal to perform navigation for the object wearing the smart wearable device.

[0009] According to another aspect of the present invention, a navigation device for the blind based on visual-tactile cross-modal perception is provided. The device comprises:

[0010] An image acquisition module, used to acquire a first environment image taken by a first camera of the smart glasses, acquire a second environment image taken by a second camera of the smart glasses, and acquire a target depth image based on the first environment image and the second environment image;

[0011] A signal conversion module, used to divide the target depth image into at least two regional images, determine the minimum depth value of each of the regional images, and convert each of the minimum depth values ​​into a voltage signal of corresponding strength based on a preset corresponding relationship between the depth value size and the voltage signal strength, wherein the regional image corresponds to the vibration unit;

[0012] The navigation module is used to transmit each of the voltage signals to the smart belt, drive the vibration unit corresponding to the voltage signal to vibrate at the intensity of the voltage signal, so as to navigate the object wearing the smart wearable device.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the blind navigation method based on visual-tactile cross-modal perception as described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the blind navigation method based on visual-tactile cross-modal perception as described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention is to obtain a first environment image taken by a first camera based on smart glasses and obtain a second environment image taken by a second camera based on the smart glasses. In this technical solution, the smart glasses equipped with dual cameras can capture information about the surrounding environment in real time to obtain a first environment image and a second environment image. Afterwards, a target depth image is obtained based on the first environment image and the second environment image to provide more accurate spatial information. The target depth image is divided into at least two regional images, the minimum depth value of each regional image is determined, and each minimum depth value is converted into a voltage signal of corresponding intensity based on a preset correspondence between the depth value size and the voltage signal intensity, wherein the regional image corresponds to the vibration unit. Each of the voltage signals is transmitted to the smart belt, and the vibration unit corresponding to the voltage signal is driven to vibrate with the intensity of the voltage signal to navigate for the object wearing the smart wearable device. The technical solution of the embodiment of the present invention realizes a precise conversion from vision to touch through a multi-point vibration design, providing a cross-modal navigation experience for users.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart of a blind navigation method based on visual-tactile cross-modal perception provided by an embodiment of the present invention;

[0022] Figure 2 A flowchart of a blind navigation method based on visual-tactile cross-modal perception provided by an embodiment of the present invention;

[0023] Figure 3 An intelligent wearable device suitable for a blind person navigation method based on visual-tactile cross-modal perception provided by an embodiment of the present invention;

[0024] Figure 4 A flowchart of an optional embodiment of a navigation method for the blind based on visual-tactile cross-modal perception provided by an embodiment of the present invention;

[0025] Figure 5A schematic diagram of an image processing flow applicable to a blind navigation method based on visual-tactile cross-modal perception provided by an embodiment of the present invention;

[0026] Figure 6 A schematic structural diagram of a navigation device for the blind based on visual-tactile cross-modal perception provided by an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0031] Figure 1 A flow chart of a blind navigation method based on visual-touch cross-modal perception is provided in an embodiment of the present invention. This embodiment is applicable to the situation of providing navigation for the blind. The method can be performed by a blind navigation device based on visual-touch cross-modal perception. The blind navigation device based on visual-touch cross-modal perception can be implemented in the form of hardware and / or software. The blind navigation device based on visual-touch cross-modal perception can be configured in an electronic device such as a computer or a server.

[0032] like Figure 1 As shown, the method of this embodiment includes:

[0033] S110, obtaining a first environment image taken by a first camera of the smart glasses, obtaining a second environment image taken by a second camera of the smart glasses, and obtaining a target depth image based on the first environment image and the second environment image.

[0034] In an embodiment of the present invention, the smart glasses may include a first camera and a second camera. The first camera and the second camera are respectively located at the left and right lens positions of the smart glasses. Optionally, the first camera may be located at the center of the left lens of the smart glasses, and the second camera may be located at the center of the right lens of the smart glasses. The first environment image may be understood as an environment image collected by the first camera. The second environment image may be understood as an environment image collected by the second camera. Optionally, the first camera and the second camera are RGB cameras, respectively. Correspondingly, the first environment image may be an RGB color image, and the second environment image data may be an RGB color image. In an embodiment of the present invention, the target depth image may be understood as a depth image for dividing the area. Specifically, a first environment image captured by the first camera of the smart glasses is obtained, and a second environment image captured by the second camera of the smart glasses is obtained. After the first environment image and the second environment image are obtained, a depth image generation algorithm may be used to generate a depth image based on the first environment image and the second environment image. Thus, the generated depth image may be used as a target depth image.

[0035] In an embodiment of the present invention, before obtaining the target depth image based on the first environmental image and the second environmental image, the method may further include: in the case where the first environmental image has image distortion, the first environmental image may be subjected to image distortion correction processing; and / or, in the case where the second environmental image has image distortion, the second environmental image may be subjected to image distortion correction processing. In an embodiment of the present invention, the image distortion correction processing for the first environmental image may be performed by using the distortion coefficient in the camera intrinsic parameter of the first camera to perform image distortion correction processing on the first environmental image. The image distortion correction processing for the second environmental image may be performed by using the distortion coefficient in the camera intrinsic parameter of the second camera to perform distortion correction processing on the second environmental image.

[0036] In the embodiment of the present invention, the purpose of using the distortion coefficients in the camera intrinsic parameters to perform distortion correction processing on the first environment image and the second environment image is to eliminate the influence of camera distortion on the image. It should be noted that the camera intrinsic parameters of the first camera and the camera intrinsic parameters of the second camera are the same.

[0037] S120, dividing the target depth image into at least two area images, determining the minimum depth value of each of the area images, and converting each of the minimum depth values ​​into a voltage signal of corresponding intensity based on a preset correspondence between the depth value size and the voltage signal intensity, wherein the area image corresponds to the vibration unit.

[0038] Among them, the regional image can be an image obtained after dividing the target depth image according to actual needs or preset rules. It can be understood that different regional images correspond to different areas. The number of regional images can be two or more. In practical applications, the number of regional images is usually multiple. Optionally, the regional image can be a fixed geometric shape, such as a rectangle, a circle, etc. In an embodiment of the invention, the target depth image can be divided according to the specification of M×N, where M can be represented by the number of rows of each divided area, and N can be represented by the number of columns of each divided area. The minimum depth value can be the minimum value of the depth values ​​of all pixels in each divided regional image. The depth value can represent the distance from the surface of an object in the scene to the first camera and / or the second camera, so the minimum depth value can represent the distance of the surface of the object (such as an obstacle) closest to the first camera and / or the second camera in the area. In an embodiment of the present invention, the minimum depth value of each of the regional images is determined, and specifically, for each of the regional images, the minimum depth value of the regional image is determined according to the depth information of the regional image.

[0039] In an embodiment of the present invention, the preset correspondence between the depth value and the voltage signal strength may be a linear relationship or a nonlinear relationship. In an embodiment of the invention, the preset correspondence between the depth value and the voltage signal strength may also determine the most suitable conversion curve based on the sensitivity of human perception. In an embodiment of the present invention, the regional image corresponds to the vibration unit. In other words, the correspondence between the regional image and the vibration unit may be one-to-one. In an embodiment of the present invention, the advantage of one regional image corresponding to one vibration unit is that the accuracy of navigation can be improved. Optionally, the vibration unit may be a vibration motor.

[0040] Specifically, after obtaining the target depth image, the target depth image can be divided into at least two regional images according to a preset rule. Thus, for each regional image, the minimum depth value of the regional image can be determined. Furthermore, based on the preset corresponding relationship between the depth value size and the voltage signal strength, each of the minimum depth values ​​can be converted into a voltage signal of corresponding strength.

[0041] S130: Transmit each of the voltage signals to the smart belt, and drive a vibration unit corresponding to the voltage signal to vibrate at the intensity of the voltage signal to perform navigation for the object wearing the smart wearable device.

[0042] In an embodiment of the present invention, the smart belt may be a flexible smart belt. The smart belt includes at least two vibration units. Optionally, the vibration unit may include a vibration motor and / or a vibration sheet. In practical applications, the smart belt generally includes a plurality of vibration units. In an embodiment of the present invention, a plurality of vibration units are evenly distributed on the smart belt. For example, the smart belt is divided according to the rule of H×W, and a vibration unit may be provided for each area, where H may be expressed as the number of rows of each divided area, and W may be expressed as the number of columns of each divided area. In an embodiment of the present invention, the correspondence between the regional image obtained by dividing the target depth image and the region obtained after the division of the smart belt may be one-to-one. In an embodiment of the present invention, the smart belt converts the information in the depth image into an electrical signal through a built-in vibration unit, and drives the vibration unit to vibrate with different intensities, thereby providing intuitive navigation information to the wearer.

[0043] Specifically, after obtaining each voltage signal, each voltage signal can be transmitted to the smart belt. For each voltage signal, a vibration unit corresponding to the voltage signal can be determined. Then, the vibration unit can be driven to vibrate with the intensity of the voltage signal to navigate the object wearing the smart wearable device. Among them, determining the vibration unit corresponding to the voltage signal can specifically include first determining a regional image containing a minimum depth value corresponding to the voltage signal. Afterwards, the vibration unit corresponding to the regional image can be used as the vibration unit corresponding to the voltage signal.

[0044] In an embodiment of the present invention, the vibration unit vibrates at different intensities according to the minimum depth value of each area image. The closer the object is, the greater the vibration intensity. This navigation method enables the user to perceive the distance and direction of surrounding objects through different positions of the waist. It is understandable that the stronger the vibration, the more potential danger there is in that direction. The user can adjust the direction of travel in time according to the vibration feedback, thereby realizing the cross-modal conversion from visual information to tactile feedback, helping the user to perceive changes in the surrounding environment through touch without voice prompts, and respond, thereby improving the accuracy of navigation and obstacle avoidance.

[0045] It can be understood that in the embodiment of the invention, when the depth value is smaller, the voltage signal converted based on the depth value may be larger. When the voltage signal is larger, the intensity of the vibration of the vibration unit is stronger, which can indicate that the distance to the object is closer. Conversely, when the depth value is larger, the voltage signal converted from the larger depth value may be smaller. When the voltage signal is smaller, the vibration intensity of the vibration unit is weaker, which can indicate that the distance to the object is farther.

[0046] In an embodiment of the present invention, before obtaining a first environment image captured by a first camera of the smart glasses and obtaining a second environment image captured by a second camera of the smart glasses, the method further includes: performing camera calibration between the first camera and / or the second camera to obtain internal camera parameters.

[0047] In an embodiment of the present invention, the camera internal parameters may include at least focal length, optical center position, and distortion coefficient. It should be noted that the camera internal parameters of the first camera and the second camera are the same. The camera internal parameters may be the camera internal parameters of the first camera or the camera internal parameters of the second camera. In an embodiment of the present invention, a preset calibration algorithm may be used to calibrate the first camera. The preset calibration algorithm may be set according to actual needs, which is not specifically limited here, for example, the Zhang Zhengyou calibration algorithm, the Tsai-Lenz (Tsai) calibration algorithm, and the like.

[0048] The technical solution of the embodiment of the present invention is to obtain a first environment image taken by a first camera based on smart glasses and obtain a second environment image taken by a second camera based on the smart glasses. In this technical solution, the smart glasses equipped with dual cameras can capture information about the surrounding environment in real time to obtain a first environment image and a second environment image. Afterwards, a target depth image is obtained based on the first environment image and the second environment image to provide more accurate spatial information. The target depth image is divided into at least two regional images, the minimum depth value of each regional image is determined, and each minimum depth value is converted into a voltage signal of corresponding intensity based on a preset correspondence between the depth value size and the voltage signal intensity, wherein the regional image corresponds to the vibration unit. Each of the voltage signals is transmitted to the smart belt, and the vibration unit corresponding to the voltage signal is driven to vibrate with the intensity of the voltage signal to navigate for the object wearing the smart wearable device. The technical solution of the embodiment of the present invention realizes a precise conversion from vision to touch through a multi-point vibration design, providing a cross-modal navigation experience for users.

[0049] Figure 2A flowchart of a blind navigation method based on visual-tactile cross-modal perception provided by an embodiment of the present invention, based on the above-mentioned embodiment, optionally, the target depth image is obtained based on the first environment image and the second environment image, including: obtaining an initial depth image based on the first environment image and the second environment image; when the depth information of the initial depth image is incomplete, the depth information of the initial depth image is completed, and the completed image is used as the target depth image; when the depth information of the initial depth image is complete, the initial depth image is used as the target depth image. Among them, the technical features that are the same or similar to those in the above-mentioned embodiment are not repeated here.

[0050] like Figure 2 As shown, the method of this embodiment specifically includes:

[0051] S210: Acquire a first environment image captured by a first camera of the smart glasses, and acquire a second environment image captured by a second camera of the smart glasses.

[0052] S220: Obtain an initial depth image based on the first environment image and the second environment image.

[0053] The initial depth image may be understood as a depth image obtained based on the first environment image and the second environment image.

[0054] In an embodiment of the present invention, there are two ways to obtain an initial depth image based on the first environment image and the second environment image. As an optional implementation of an embodiment of the present invention, a deep learning algorithm (e.g., MonoDepth algorithm, StereoNet algorithm) can be used to obtain an initial depth image based on the first environment image and the second environment image. In an embodiment of the present invention, a more refined depth image can be preset from a binocular image through a neural network using a deep learning algorithm.

[0055] As another optional implementation of an embodiment of the present invention, a Semi-Global BlockMatching (SGBM) algorithm may be used to generate a depth image based on the first environmental image and the second environmental image. Thus, the generated depth image may be used as an initial depth image. Specifically, the SGBM algorithm may be used to calculate the view difference between the first environmental image and the second environmental image. The view difference may then be converted into a depth map, i.e., an initial depth image. In an embodiment of the present invention, the benefit of using the SGBM algorithm to generate a depth image is that it can achieve a good balance between real-time and precision, so that a more accurate depth image can be obtained more quickly.

[0056] S230: When the depth information of the initial depth image is incomplete, perform depth information completion processing on the initial depth image, and use the completed image as a target depth image.

[0057] In an embodiment of the present invention, when the depth information of the initial depth image is incomplete, the depth information of the initial depth image can be completed based on a preset depth completion algorithm, and the completed image is used as the target depth image. Among them, the preset depth completion algorithm is set according to actual needs, and it is not specifically limited here, for example, AGG-Net (Attention Guided Gated-convolutional Network) based on deep learning. In an embodiment of the present invention, the use of the preset depth completion algorithm can generate a depth image with more complete and accurate depth information when the depth information of the initial depth image is incomplete, thereby providing a basis for the generation of subsequent signals, thereby improving the accuracy of navigation.

[0058] On the basis of the above embodiment, when the depth information of the initial depth image is sparse, the depth information may be completed for the initial depth image, and the completed image may be used as the target depth image.

[0059] S240: When the depth information of the initial depth image is complete, use the initial depth image as a target depth image.

[0060] Specifically, when the depth information of the initial depth image is complete, the initial depth image may be used as the target depth image.

[0061] S250, dividing the target depth image into at least two area images, determining the minimum depth value of each of the area images, and converting each of the minimum depth values ​​into a voltage signal of corresponding intensity based on a preset correspondence between the depth value size and the voltage signal strength, wherein the area image corresponds to the vibration unit.

[0062] S260: Transmit each of the voltage signals to the smart belt, and drive a vibration unit corresponding to the voltage signal to vibrate at the intensity of the voltage signal to perform navigation for the object wearing the smart wearable device.

[0063] The technical solution of the embodiment of the present invention is to obtain an initial depth image based on the first environment image and the second environment image; when the depth information of the initial depth image is incomplete, the depth information of the initial depth image is completed, and the completed image is used as the target depth image; when the depth information of the initial depth image is complete, the initial depth image is used as the target depth image, so as to obtain a depth image with more complete and accurate depth information, thereby improving the accuracy of navigation.

[0064] The embodiment of the present invention provides an optional embodiment of a blind navigation method based on visual-tactile cross-modal perception. The smart wearable device of the embodiment of the present invention may include smart glasses ( Figure 3 1) and smart belt ( Figure 3 2), wherein the smart glasses may include two RGB cameras ( Figure 3 3), the smart belt includes a plurality of vibration plates ( Figure 3 4). Figure 4 According to the embodiment, the navigation for the blind based on the smart wearable device specifically includes the following steps:

[0065] First, the two cameras of the smart glasses collect environmental images, i.e., GRB color images. After that, the collected GRB color images are processed to obtain the target depth image, and the target depth image is segmented to obtain the closest point, which is the closest point depth map, wherein the closest point depth map includes the minimum depth value of each segmented area. Then, the minimum depth value of each area can be converted into a voltage signal, thereby driving the corresponding vibration plate on the smart belt to vibrate with different intensities, so that the user can perceive the distance and direction of surrounding objects through different positions of the waist.

[0066] In the embodiment of the present invention, Figure 5 As shown, the collected GRB color image is processed to obtain a target depth image, and the target depth image is segmented to find the closest point. The specific steps may include: first, binocular positioning and camera internal parameter conversion can be performed. After that, the left and right images, i.e., the first environment image and the second environment image, can be collected in real time by the RGB camera of the smart glasses. After the left and right images are collected, image distortion correction can be performed. After that, the SGBM algorithm can be used to process the distortion-corrected left and right images to generate an initial depth image. After obtaining the initial depth image, the AGG-Net algorithm can be used to complete the depth information of the initial depth image to obtain the target depth image. Then, the target depth image can be segmented to obtain the regional images corresponding to each region. Thereby, the distance value of the nearest point, i.e., the minimum depth value of each regional image, can be calculated for each region.

[0067] The technical solution of the embodiment of the present invention is to obtain a first environment image taken by a first camera based on smart glasses and obtain a second environment image taken by a second camera based on the smart glasses. In this technical solution, the smart glasses equipped with dual cameras can capture information about the surrounding environment in real time to obtain a first environment image and a second environment image. Afterwards, a target depth image is obtained based on the first environment image and the second environment image to provide more accurate spatial information. The target depth image is divided into at least two regional images, the minimum depth value of each regional image is determined, and each minimum depth value is converted into a voltage signal of corresponding intensity based on a preset correspondence between the depth value size and the voltage signal intensity, wherein the regional image corresponds to the vibration unit. Each of the voltage signals is transmitted to the smart belt, and the vibration unit corresponding to the voltage signal is driven to vibrate with the intensity of the voltage signal to navigate for the object wearing the smart wearable device. The technical solution of the embodiment of the present invention realizes a precise conversion from vision to touch through a multi-point vibration design, providing a cross-modal navigation experience for users.

[0068] Figure 6 The structure diagram of a blind navigation device based on visual-tactile cross-modal perception provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the device includes: an image acquisition module 310, a signal conversion module 320 and a navigation module 330.

[0069] Among them, the image acquisition module 310 is used to obtain a first environment image taken by a first camera based on the smart glasses, obtain a second environment image taken by a second camera based on the smart glasses, and obtain a target depth image based on the first environment image and the second environment image; the signal conversion module 320 is used to divide the target depth image into at least two regional images, determine the minimum depth value of each of the regional images, and convert each of the minimum depth values ​​into a voltage signal of corresponding intensity based on a preset correspondence between the depth value size and the voltage signal intensity, wherein the regional image corresponds to the vibration unit; the navigation module 330 is used to transmit each of the voltage signals to the smart belt, drive the vibration unit corresponding to the voltage signal to vibrate with the intensity of the voltage signal, so as to navigate the object wearing the smart wearable device.

[0070] The technical solution of the embodiment of the present invention is to obtain a first environment image taken by a first camera based on smart glasses through an image acquisition module, and obtain a second environment image taken by a second camera based on the smart glasses. In this technical solution, the smart glasses equipped with dual cameras can capture information about the surrounding environment in real time to obtain a first environment image and a second environment image. Afterwards, a target depth image is obtained based on the first environment image and the second environment image through a signal conversion module to provide more accurate spatial information. The target depth image is divided into at least two regional images, the minimum depth value of each regional image is determined, and each minimum depth value is converted into a voltage signal of corresponding intensity based on a preset correspondence between the depth value size and the voltage signal intensity, wherein the regional image corresponds to the vibration unit. Each of the voltage signals is transmitted to the smart belt through a navigation module, and the vibration unit corresponding to the voltage signal is driven to vibrate with the intensity of the voltage signal to navigate for the object wearing the smart wearable device. The technical solution of the embodiment of the present invention realizes a precise conversion from vision to touch through a multi-point vibration design, providing a cross-modal navigation experience for users.

[0071] Optionally, the signal conversion module includes a depth image generation unit, wherein the depth image generation unit is used to obtain an initial depth image based on the first environmental image and the second environmental image; when the depth information of the initial depth image is incomplete, the depth information of the initial depth image is completed, and the completed image is used as the target depth image; when the depth information of the initial depth image is complete, the initial depth image is used as the target depth image.

[0072] Optionally, the depth image generating unit is configured to obtain an initial depth image based on the first environment image and the second environment image by adopting an SGBM algorithm.

[0073] Optionally, the device also includes a camera calibration module, which is used to perform camera calibration between the first camera and / or the second camera to obtain camera internal parameters before obtaining the first environment image taken by the first camera of the smart glasses and obtaining the second environment image taken by the second camera of the smart glasses.

[0074] Optionally, the first camera and the second camera are RGB cameras respectively.

[0075] Optionally, the device also includes an image distortion correction module, wherein the image distortion correction module is used to, before obtaining the target depth image based on the first environmental image and the second environmental image, perform image distortion correction processing on the first environmental image in the case that image distortion occurs in the first environmental image; and / or, when image distortion occurs in the second environmental image, perform image distortion correction processing on the second environmental image.

[0076] Optionally, the preset corresponding relationship between the depth value and the voltage signal strength may be a linear relationship or a non-linear relationship.

[0077] The blind navigation device based on visual-tactile cross-modal perception provided by the embodiment of the present invention can execute the blind navigation method based on visual-tactile cross-modal perception provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0078] It is worth noting that the various units and modules included in the above-mentioned blind navigation device based on visual-touch cross-modal perception are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0079] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0080] like Figure 7As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0081] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0082] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a navigation method for the blind based on visual-tactile cross-modal perception.

[0083] In some embodiments, the blind navigation method based on visual-tactile cross-modal perception may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the blind navigation method based on visual-tactile cross-modal perception described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the blind navigation method based on visual-tactile cross-modal perception in any other appropriate manner (e.g., by means of firmware).

[0084] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0085] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0086] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0087] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0088] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0089] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0090] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0091] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A navigation method for the blind based on visual-tactile cross-modal perception, characterized in that: The method is executed by a processor of a smart wearable device, the smart wearable device includes smart glasses and a smart belt, the smart belt is provided with at least two vibration units, and the method includes: Acquire a first environment image taken by a first camera of the smart glasses, acquire a second environment image taken by a second camera of the smart glasses, and obtain a target depth image based on the first environment image and the second environment image; Dividing the target depth image into at least two regional images, determining a minimum depth value of each of the regional images, and converting each of the minimum depth values ​​into a voltage signal of a corresponding strength based on a preset corresponding relationship between the depth value size and the voltage signal strength, wherein the regional image corresponds to the vibration unit; Each of the voltage signals is transmitted to the smart belt, and a vibration unit corresponding to the voltage signal is driven to vibrate at the intensity of the voltage signal to perform navigation for the object wearing the smart wearable device.

2. The method according to claim 1, characterized in that The obtaining a target depth image based on the first environment image and the second environment image includes: Obtaining an initial depth image based on the first environment image and the second environment image; In the case where the depth information of the initial depth image is incomplete, performing depth information completion processing on the initial depth image, and using the completed image as the target depth image; When the depth information of the initial depth image is complete, the initial depth image is used as the target depth image.

3. The method according to claim 2, characterized in that The obtaining an initial depth image based on the first environment image and the second environment image includes: An initial depth image is obtained based on the first environment image and the second environment image by using the SGBM algorithm.

4. The method according to claim 1, characterized in that: Before acquiring the first environment image captured by the first camera of the smart glasses and acquiring the second environment image captured by the second camera of the smart glasses, the method further includes: Perform camera calibration on the first camera and / or the second camera to obtain camera internal parameters.

5. The method according to claim 1, characterized in that The first camera and the second camera are RGB cameras respectively.

6. The method according to claim 1, characterized in that Before obtaining the target depth image based on the first environment image and the second environment image, the method further includes: In the case where the first environmental image has image distortion, image distortion correction processing is performed on the first environmental image; and / or, in the case where the second environmental image has image distortion, image distortion correction processing is performed on the second environmental image.

7. The method according to claim 1, characterized in that The preset corresponding relationship between the depth value and the voltage signal strength may be a linear relationship or a non-linear relationship.

8. A navigation device for the blind based on visual-tactile cross-modal perception, characterized in that: The device comprises: An image acquisition module, used to acquire a first environment image taken by a first camera of the smart glasses, acquire a second environment image taken by a second camera of the smart glasses, and acquire a target depth image based on the first environment image and the second environment image; A signal conversion module, used to divide the target depth image into at least two regional images, determine the minimum depth value of each of the regional images, and convert each of the minimum depth values ​​into a voltage signal of corresponding strength based on a preset corresponding relationship between the depth value size and the voltage signal strength, wherein the regional image corresponds to the vibration unit; The navigation module is used to transmit each of the voltage signals to the smart belt, drive the vibration unit corresponding to the voltage signal to vibrate at the intensity of the voltage signal, so as to navigate the object wearing the smart wearable device.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the blind navigation method based on visual-tactile cross-modal perception as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the blind navigation method based on visual-tactile cross-modal perception according to any one of claims 1 to 7 when executed.

Images