A camera and ToF module-based obstacle avoidance navigation system and method

Abstract

The application relates to the technical field of obstacle avoidance navigation, in particular to an obstacle avoidance navigation system and method based on a camera and a ToF module, which comprises the following modules: a camera module, which is used for collecting a scene image and outputting the scene image; a ToF module, which is used for acquiring a scene depth map and outputting the scene depth map; and a target identification module, which is used for detecting the scene image by using a convolutional neural network to obtain a key target. In the application, the target is clear, only the key target is identified, the training data requirement and the calculation complexity are reduced, and the application is suitable for running on a mobile phone or an embedded device; meanwhile, vision provides rich semantic information, ToF provides reliable depth data, the two are fused to improve the robustness of the system in a complex environment such as weak texture and strong light, and through grid processing and a multi-frame fusion strategy, low delay and high reliability of the obstacle avoidance prompt are realized under limited computing power.

Description

Technical Field

[0001] This invention relates to the field of obstacle avoidance navigation technology, and in particular to an obstacle avoidance navigation system and method based on a camera and a ToF module. Background Technology

[0002] With the development of computer vision, sensor fusion, and embedded artificial intelligence technologies, assistive devices for the blind are gradually evolving towards intelligence, lightweight design, and practicality. Currently available obstacle avoidance solutions for the blind mainly fall into the following categories:

[0003] 1. Pure Vision Solution: This solution uses monocular, binocular, or multi-view cameras to perform obstacle detection and depth estimation through stereo matching or deep learning. This type of solution performs well in environments with rich textures and stable lighting, but suffers from large depth estimation errors and real-time performance issues under conditions of weak textures, strong light, or low light.

[0004] 2. Pure ToF / Radar Solution: This solution uses ToF sensors or millimeter-wave radar to directly acquire depth information. It has the advantages of accurate ranging and being unaffected by lighting conditions. However, it has low spatial resolution, lacks semantic information, and is difficult to identify specific types of targets (such as blind paths, traffic lights, etc.).

[0005] 3. Multi-sensor fusion solution: Although the visual and ultrasonic fusion solution improves reliability, the ultrasonic sensor has a limited detection range and poor directionality, which cannot meet the needs of fine navigation in complex scenarios.

[0006] In summary, no existing technology has yet provided a lightweight system capable of achieving key target identification and reliable obstacle avoidance in complex scenarios while maintaining real-time performance and low cost. Therefore, this invention proposes a vision-ToF (Time-of-Flight) deep fusion obstacle avoidance and navigation system. By defining and identifying key targets and fusing visual semantics with ToF depth information, this system significantly reduces algorithm complexity and hardware costs while ensuring real-time performance and reliability. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and propose an obstacle avoidance navigation system and method based on a camera and a ToF module.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: an obstacle avoidance navigation system based on a camera and a ToF module includes:

[0009] The camera module is used to perform the steps of acquiring scene images and outputting the scene images;

[0010] The ToF module is used to perform the steps of acquiring a scene depth map and outputting the scene depth map.

[0011] The target recognition module is used to perform the step of detecting key targets in the scene image using a convolutional neural network;

[0012] The data fusion module is used to perform the step of mapping the key target and the scene depth map to a spatial grid to obtain the grid mapping result;

[0013] The obstacle determination module is used to perform the step of determining obstacles based on the grid mapping results and obtaining the determination results;

[0014] The navigation prompt module is used to perform the steps of generating a navigation table based on the determination result and outputting voice prompts;

[0015] The communication coordination module is used to perform the steps of establishing a communication connection with the smart terminal and synchronously outputting the voice prompts.

[0016] Preferably, the steps of acquiring and outputting the scene image are as follows:

[0017] The system is equipped with a wide-angle camera integrated into the smart glasses. The wide-angle camera is used to capture scene images. The frame rate of the wide-angle camera is adjusted to a preset frame rate. The captured scene images are compressed in JPEG format. The compressed scene images are then transmitted to a processing terminal and output as scene images.

[0018] Preferably, the steps for obtaining and outputting the scene depth map are as follows:

[0019] A planar array sensor is configured and installed parallel to the optical axis of the wide-angle camera. The resolution of the planar array sensor is set to a preset resolution. The planar array sensor is used to acquire a scene depth map. The acquisition frame rate of the planar array sensor is controlled to be synchronized with the frame rate of the wide-angle camera. The scene depth map is filtered. The filtered scene depth map is then aligned with coordinates and output.

[0020] Preferably, the step of using a convolutional neural network to detect key targets in the scene image specifically includes:

[0021] A convolutional neural network is deployed in the processing terminal. The scene image is input into the convolutional neural network, and the semantic features of the scene image are extracted using the convolutional neural network. Multiple preset categories of targets in the semantic features are identified, and the multiple preset categories of targets are summarized to obtain key targets. The preset categories of targets include obstacle targets and road guidance targets.

[0022] Preferably, the step of mapping the key target and the scene depth map to a spatial grid to obtain the grid mapping result is as follows:

[0023] The field of view is divided into an M x M spatial grid. Each grid in the spatial grid corresponds to a preset physical size. The position coordinates of the key target are extracted and mapped to the corresponding grid position in the spatial grid. A visual recognition state label is assigned to the corresponding grid position to obtain the visual recognition state corresponding to the spatial grid. The depth point cloud data of the scene depth map is extracted and mapped to the corresponding grid position in the spatial grid. A depth detection state label is assigned to the corresponding grid position to obtain the depth detection state corresponding to the spatial grid. The visual recognition state and the depth detection state in the spatial grid are integrated to obtain the grid mapping result.

[0024] Preferably, the specific steps for determining obstacles based on the mesh mapping results and obtaining the determination results are as follows:

[0025] Extract the visual recognition state and the depth detection state of each grid in the grid mapping result, and determine the visual recognition state and the depth detection state;

[0026] If the visual recognition state is an obstacle state and the depth detection state is an obstacle state, the grid is confirmed to be impassable, and a judgment result is obtained.

[0027] If the visual recognition state is an obstacle state and the depth detection state is a non-obstacle state, extract the depth distance data corresponding to the depth detection state, and determine whether the depth distance data corresponding to the depth detection state is less than or equal to a preset distance threshold. If the depth distance data corresponding to the depth detection state is less than or equal to the preset distance threshold, confirm that the grid is in an avoidance state. If the depth distance data corresponding to the depth detection state is greater than the preset distance threshold, confirm that the grid is in a pending confirmation state, and obtain the determination result.

[0028] If the visual recognition state is a non-obstacle state and the depth detection state is an obstacle state, extract the depth distance data corresponding to the depth detection state, and determine whether the depth distance data corresponding to the depth detection state is less than or equal to the preset distance threshold. If the depth distance data corresponding to the depth detection state is less than or equal to the preset distance threshold, confirm that the grid is in an avoidance state. If the depth distance data corresponding to the depth detection state is greater than the preset distance threshold, confirm that the grid is in a pending confirmation state, and obtain the determination result.

[0029] If the visual recognition state is a non-obstacle state and the depth detection state is a non-obstacle state, the grid is confirmed to be passable, and a judgment result is obtained.

[0030] Preferably, the steps of generating a navigation table based on the determination result and outputting voice prompts are as follows:

[0031] Extract the impassable state, avoidance state, and passable state from the determination result. Generate a navigation table of preset dimensions based on the impassable state, avoidance state, and passable state. Locate the user's current position in the navigation table. Expand to the left by a preset number of steps from the current position in the navigation table. Calculate the number of consecutive passable grids corresponding to the left side. Expand to the right by the preset number of steps from the current position in the navigation table. Calculate the number of consecutive passable grids corresponding to the right side. Compare the number of consecutive passable grids corresponding to the left side and the number of consecutive passable grids corresponding to the right side. Filter the direction corresponding to the maximum value of the consecutive passable grids. Generate one of the left-side passable prompt sound and the right-side passable prompt sound based on the direction corresponding to the maximum value, and output the voice prompt.

[0032] Preferably, the steps for establishing a communication connection with the smart terminal and synchronously outputting the voice prompts are as follows:

[0033] A Bluetooth communication connection is established between the processing terminal and the smart terminal. The voice prompt is sent to the navigation application installed inside the smart terminal. The bone conduction headphones are invoked to play the voice prompt received by the navigation application. The voice prompt is output synchronously in conjunction with the path guidance information of the navigation application.

[0034] The present invention also provides a method comprising the following steps:

[0035] Acquire scene images and output the scene images;

[0036] Obtain the scene depth map and output the scene depth map;

[0037] The key targets are obtained by detecting the scene image using a convolutional neural network;

[0038] The key targets are mapped to the scene depth map onto a spatial grid to obtain the grid mapping result;

[0039] Obstacles are determined based on the grid mapping results, and the determination results are obtained.

[0040] A navigation table is generated based on the determination result, and voice prompts are output.

[0041] Establish a communication connection with the smart terminal and output the voice prompts synchronously.

[0042] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0043] 1. Clear target identification and lightweight algorithm: It only identifies key targets, reducing the need for training data and computational complexity, making it suitable for running on mobile phones or embedded devices.

[0044] 2. Complementary sensor advantages: Vision provides rich semantic information, while ToF provides reliable depth data. The fusion of the two enhances the system's robustness in complex environments such as weak texture and strong light.

[0045] 3. Controllable cost: It adopts consumer-grade cameras and ToF modules, eliminating the need for high-cost LiDAR, which is conducive to product popularization.

[0046] 4. Balancing real-time performance and practicality: Through grid-based processing and multi-frame fusion strategies, low-latency and highly reliable obstacle avoidance prompts are achieved with limited computing power. Attached Figure Description

[0047] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0049] Please see Figure 1 This invention provides a technical solution: an obstacle avoidance navigation system based on a camera and a ToF module, comprising:

[0050] The camera module is used to perform the steps of acquiring scene images and outputting scene images;

[0051] The ToF module is used to perform the steps of acquiring and outputting the scene depth map.

[0052] The target recognition module is used to perform the step of detecting key targets in scene images using a convolutional neural network;

[0053] The data fusion module is used to perform the steps of mapping key targets and scene depth maps to a spatial grid to obtain the grid mapping results;

[0054] The obstacle determination module is used to perform the step of determining obstacles based on the mesh mapping results and obtaining the determination results;

[0055] The navigation prompt module is used to perform the steps of generating a navigation table based on the judgment result and outputting voice prompts;

[0056] The communication coordination module is used to perform the steps of establishing a communication connection with the smart terminal and synchronously outputting voice prompts.

[0057] In this embodiment, the steps of acquiring and outputting scene images are as follows: configure the wide-angle camera integrated into the smart glasses, acquire scene images using the wide-angle camera, adjust the frame rate of the wide-angle camera to a preset frame rate, compress the acquired scene images in JPEG format, transmit the compressed scene images to the processing terminal, and output the scene images.

[0058] Specifically, the process of acquiring scene images is as follows: A wide-angle camera fixed to the frame of the smart glasses is configured; the image acquisition interface of the wide-angle camera is activated to acquire scene images containing continuous optical signals of the foreground environment; the sampling frequency of the wide-angle camera is adjusted to a preset frame rate. The preset frame rate depends on the real-time walking speed parameters. The real-time walking speed is obtained by reading the average acceleration over three consecutive seconds output by the inertial measurement unit and performing an integral operation over time. The real-time walking speed is then compared with a pre-written one-dimensional comparison table in memory. For example, when the real-time walking speed is between 0.5 meters per second and 1.2 meters per second, the preset frame rate is set to 10 frames per second; when the real-time walking speed is between 1.2 meters per second and 2.0 meters per second, the preset frame rate is set to 20 frames per second. If the distance exceeds 2.0 meters per second, the frame rate is set to 30 frames per second. Following this rule, the preset frame rate for the current moment is locked. The acquired scene image is converted to a color space for luminance and chromaticity channels. The luminance and chromaticity components of the scene image are extracted. The scene image is divided into 8x8 pixel two-dimensional matrix blocks. A discrete cosine transform is performed on each two-dimensional matrix block to convert the pixel values ​​in the spatial domain into transform coefficients in the frequency domain. The transform coefficients are discretized and scaled using a quantization table. The two-dimensional matrix is ​​converted into a one-dimensional array using a zigzag scan. The one-dimensional array is entropy encoded using a Huffman tree structure to form a compressed binary bitstream in JPEG format. The scene image in compressed binary bitstream form is sent to a portable processing terminal via a universal serial bus interface and temporarily stored in the random access memory of the processing terminal, completing the output of the scene image.

[0059] In this embodiment, the steps for acquiring and outputting the scene depth map are as follows: configuring an area array sensor installed parallel to the optical axis of the wide-angle camera, setting the resolution of the area array sensor to a preset resolution, acquiring the scene depth map using the area array sensor, controlling the acquisition frame rate of the area array sensor to synchronize with the frame rate of the wide-angle camera, filtering the scene depth map, performing coordinate alignment processing on the filtered scene depth map, and outputting the scene depth map.

[0060] Specifically, the process of acquiring a scene depth map is as follows: A time-of-flight area array sensor is configured to be physically parallel to the optical axis of the wide-angle camera. The pixel array size of the area array sensor is set to a preset resolution. The preset resolution parameter is dynamically allocated by reading the remaining memory capacity of the processing terminal. The available random access memory capacity data of the processing terminal in the current cycle is extracted. For example, when the available capacity is greater than 4 gigabytes, the preset resolution is set to 640 x 480 pixels; when the available capacity is between 2 and 4 gigabytes, it is set to 320 x 240 pixels. The area array sensor is activated to emit infrared light pulses according to the determined preset resolution. Infrared photons reflected back from encountered objects are received. The time difference between the emission and reception of the light pulse is recorded. The time difference is multiplied by the speed of light and divided by 2 to obtain the depth distance value of each pixel. All depth distances are then summed. The initial scene depth map is formed by extracting the current effective frame rate value of the wide-angle camera. The trigger pulse frequency of the area array sensor is modified to be exactly equal to the frame rate value, so that the scene image and the scene depth map trigger exposure at the same time. The depth distance value of each center pixel and its surrounding 3x3 neighborhood in the initial scene depth map is read. These 9 values ​​are sorted from largest to smallest. The value of the 5th position is selected to replace the original depth distance value of the center pixel, and the median filtering of the scene depth map is completed. The intrinsic parameter matrix of the wide-angle camera and the extrinsic parameter matrix between them are extracted. The 3D coordinates of the filtered scene depth map are multiplied by the translation and rotation extrinsic parameter matrix to transform it into the camera coordinate system of the wide-angle camera. Then, it is multiplied by the intrinsic parameter matrix of the wide-angle camera and projected onto a 2D plane to generate the scene depth map data after coordinate alignment and output it.

[0061] In this embodiment, the steps of using a convolutional neural network to detect scene images and obtain key targets are as follows: deploying a convolutional neural network in a processing terminal, inputting the scene image into the convolutional neural network, using the convolutional neural network to extract semantic features of the scene image, identifying multiple preset categories of targets in the semantic features, and summarizing the multiple preset categories of targets to obtain key targets; wherein, the preset categories of targets include obstacle targets and road guidance targets.

[0062] Specifically, the key target extraction process involves deploying a multi-layer convolutional neural network within the graphics processing unit (GPU) of the processing terminal. The network structure includes an input layer, five convolutional layers with residual connections, three max-pooling layers, and two fully connected layers. The input layer's data size is set to a three-channel matrix with a width and height of 640 pixels. The output layer is a one-dimensional vector containing the target bounding box coordinates and a preset class confidence score. Training the network involves collecting 100,000 images of various urban scenes, manually labeling the bounding boxes and class labels of obstacle targets and road guidance targets, and inputting these into the multi-layer convolutional neural network for forward propagation calculations. This yields predicted bounding boxes and predicted classes. The cross-union loss between the predicted bounding boxes and the ground truth bounding boxes, as well as the cross-entropy loss between the predicted classes and the ground truth labels, are then calculated. The loss function is calculated using stochastic gradient descent with momentum, taking the partial derivative of the loss function with respect to the weights of each convolutional kernel. The initial learning rate is set to 0.01 and the momentum parameter is 0.9. The weight parameters are updated layer by layer using backpropagation. The iteration continues until the loss on the validation set fluctuates less than 0.001 for 10 consecutive cycles. The parameters are then saved to complete the training. In the inference phase, the scene image obtained earlier is extracted, scaled to 640 by 640 pixels, and fed into the input layer. The convolutional layer extracts the low-level texture and high-level semantic features, and the pooling layer downsamples the feature map to reduce the spatial dimension. The network finally outputs a one-dimensional vector containing the bounding box coordinates. Candidate boxes with a class confidence greater than 0.8 are extracted. Candidate boxes of the categories of obstacle targets and road guidance targets are merged in spatial location to generate key targets within the scene image.

[0063] In this embodiment, the specific steps for mapping key targets and scene depth maps to spatial grids to obtain grid mapping results are as follows: the field of view is divided into M-M spatial grids, each grid in the spatial grid corresponds to a preset physical size, the position coordinates of the key targets are extracted, the position coordinates of the key targets are mapped to the corresponding grid positions in the spatial grids, a visual recognition state label is assigned to the corresponding grid positions to obtain the visual recognition state corresponding to the spatial grids, the depth point cloud data of the scene depth map is extracted, the depth point cloud data of the scene depth map is mapped to the corresponding grid positions in the spatial grids, a depth detection state label is assigned to the corresponding grid positions to obtain the depth detection state corresponding to the spatial grids, and the visual recognition state and depth detection state in the spatial grids are integrated to obtain the grid mapping results.

[0064] Specifically, the process of mapping features and depth to a spatial grid involves extracting the horizontal and vertical field of view from the camera, dividing the 3D field of view into M x M equal-sized spatial grids along the horizontal and vertical directions, setting a preset physical size for each grid (this preset physical size depends on the user's average shoulder width; the standard adult shoulder width value of 0.5 meters is obtained from the settings file), setting the width of the preset physical size to 0.25 meters and the height to a uniform 0.3 meters, generating a cuboid-shaped spatial grid. The 2D pixel bounding box coordinates of each target are extracted from the previously obtained key targets. Based on the camera intrinsic parameter matrix, the 2D coordinates are inversely projected into 3D ray directions. The index numbers of all spatial grids traversed by the ray are calculated. The positions of key targets with obstacle target attributes are mapped to the corresponding spatial grids. The data bits inside the grids that are pierced by the ray are written as 1, marking them as obstacles in the visual recognition state. The data bits of the remaining grids that are not pierced are written as 0, thus obtaining the visual recognition state of the entire spatial grid. Next, the scene depth map after the coordinate alignment process is extracted. The depth point cloud data contained in each pixel in the scene depth map is traversed. For each point cloud coordinate, the specific row and column number in the spatial grid is calculated by rounding down. If the total number of depth point cloud data falling into a certain grid is greater than 50, the depth data bit of that grid is written as 1, marking the depth detection state as an object. If the number is less than or equal to 50, it remains 0, thus obtaining the depth detection state of the spatial grid. The visual recognition state and the depth detection state of each grid are concatenated with an underscore to generate a grid mapping result containing dual state encoding.

[0065] In this embodiment, the steps for obstacle determination based on the grid mapping results and obtaining the determination result are as follows: Extract the visual recognition state and depth detection state of each grid in the grid mapping results, and determine the visual recognition state and depth detection state; if both the visual recognition state and the depth detection state are obstacle states, confirm that the grid is impassable, and obtain the determination result; if both the visual recognition state and the depth detection state are not obstacle states, extract the depth distance data corresponding to the depth detection state, and determine whether the depth distance data corresponding to the depth detection state is less than or equal to a preset distance threshold; if the depth distance data corresponding to the depth detection state is less than or equal to the preset distance threshold, confirm that the grid is in an avoidance state; if the depth detection... If the depth distance data corresponding to the measured state is greater than a preset distance threshold, the grid is confirmed as a state to be confirmed, and a judgment result is obtained. If the visual recognition state is a non-obstacle state and the depth detection state is an obstacle state, the depth distance data corresponding to the depth detection state is extracted, and it is determined whether the depth distance data corresponding to the depth detection state is less than or equal to a preset distance threshold. If the depth distance data corresponding to the depth detection state is less than or equal to the preset distance threshold, the grid is confirmed as an avoidance state. If the depth distance data corresponding to the depth detection state is greater than the preset distance threshold, the grid is confirmed as a state to be confirmed, and a judgment result is obtained. If the visual recognition state is a non-obstacle state and the depth detection state is a non-obstacle state, the grid is confirmed as a passable state, and a judgment result is obtained.

[0066] Specifically, the process of performing grid state determination is as follows: It iterates through the dual state codes of the previously generated grid mapping results, splitting them into visual recognition state values ​​and depth detection state values; it reads the real-time walking speed; and calculates the preset distance threshold using a physical braking model, using the following formula: ,in, The calculated preset distance threshold is in meters. The representative real-time walking speed was set at 1.5 meters per second. This indicates that the preset user response time constant is fixed at 0.8 seconds. This indicates that the coefficient of friction between the shoe sole and the ground is set to 0.4. Assuming the gravitational acceleration constant is fixed at 9.8 m / s², the current preset distance threshold is calculated to be approximately 1.48 m. When both the visual recognition status and depth detection status of a grid are 1, an "impassable" status is written to that grid. When the visual recognition status is 1 and the depth detection status is 0, the average value of the depth point cloud data within that grid is calculated as the depth distance data. This depth distance data is compared with the calculated preset distance threshold of 1.48 m. If the depth distance data is less than or equal to 1.48 m, an "avoid" status is written to the grid; if it is greater than 1.48 m, a "pending confirmation" status is written. When the visual recognition status is 0 and the depth detection status is 1, the same method is used to calculate and compare the depth distance data. If it is less than or equal to 1.48 m, an "avoid" status is written; if it is greater than 1.48 m, a "pending confirmation" status is written. When both statuses are 0, a "passable" status is written to the grid. All statuses are packaged to obtain the judgment result.

[0067] In this embodiment, the steps of generating a navigation table based on the judgment result and outputting voice prompts are as follows: extract the impassable state, avoidable state, and passable state from the judgment result; generate a navigation table of preset dimensions based on the impassable state, avoidable state, and passable state; locate the user's current position in the navigation table; expand to the left by a preset number of steps with the current position in the navigation table as the center; calculate the number of consecutive passable grids on the left; expand to the right by a preset number of steps with the current position in the navigation table as the center; calculate the number of consecutive passable grids on the right; compare the number of consecutive passable grids on the left and the number of consecutive passable grids on the right; filter the direction corresponding to the maximum value among the consecutive passable grids; generate one of the left passable prompt sound and the right passable prompt sound based on the direction corresponding to the maximum value; and output the voice prompt.

[0068] Specifically, the process of generating the navigation table and prompts is as follows: Extract the markers for all grid cells' impassable, avoidable, and passable states from the previously obtained judgment results, and construct a two-dimensional navigation table. The number of rows and columns in this table is limited by a preset dimension, which is set based on the screen refresh rate in milliseconds of the processing terminal. For example, if the refresh rate is extracted to be 16 milliseconds, fix the preset dimension of the navigation table to a 20-row by 20-column matrix structure. Fill the grid cell status into this matrix according to the coordinates. Mark the position of the 20th row and 10th column in the navigation table as the user's current position. Using the current position as the starting point, read a preset number of steps horizontally to the left along the matrix. The value of the preset number of steps is determined by the ambient light intensity. Read the lux value of the photosensor. For example, when the light intensity... When the light intensity is below 50 lux, the preset step size is set to 3 grid units. When the light intensity is above 50 lux, it is set to 5 grid units. For example, if the current light intensity is above 50 lux, 5 grid units are read to the left. The total number of grids that are continuously displayed as passable in these 5 grids is counted and recorded as the number of consecutive passable grids on the left. Similarly, 5 grid units are read horizontally to the right from the current position. The total number of consecutive passable grids is counted and recorded as the number of consecutive passable grids on the right. The left and right statistical values ​​are input into a comparator for subtraction. The direction corresponding to the relative maximum value is selected. For example, if the left number is 4 and the right number is 2, the direction of the maximum value is to the left. The corresponding preset audio file on the left or right is called from the material library to generate a voice signal output.

[0069] In this embodiment, the steps of establishing a communication connection with the smart terminal and synchronously outputting voice prompts are as follows: establishing a Bluetooth communication connection between the processing terminal and the smart terminal, sending the voice prompts to the navigation application installed inside the smart terminal, calling the bone conduction headphones to play the voice prompts received by the navigation application, and synchronously outputting voice prompts in combination with the path guidance information of the navigation application.

[0070] Specifically, the process of establishing device communication and audio synchronization is as follows: The processing terminal's built-in Bluetooth RF chip is activated, scanning for low-power Bluetooth devices in broadcast mode. The device's Media Access Control (MAC) address is captured, and the scanned address list is matched against the registered authorized whitelist. If the device address belongs to the smart terminal, a pairing request packet is sent to that smart terminal. After exchanging public and private keys, a stable Bluetooth serial communication connection is established between the processing terminal and the smart terminal. The audio data stream generated earlier (either the left or right access prompt tone) is divided into multiple Bluetooth transmission protocol data units (512 bytes per packet) and sent to the smart terminal's operating system kernel space. The application programming interface forwards the data units to the internally running navigation application for caching and combination. The navigation application reads the latitude and longitude output by the built-in positioning module. Based on latitude and longitude data, macroscopic path guidance information to the destination is planned, such as a text instruction to turn right 200 meters from the intersection. The corresponding synthesized speech is extracted, and the navigation application schedules the audio hardware abstraction layer of the smart terminal to connect to the bone conduction headphones worn on the head. The obstacle avoidance voice prompts from the processing terminal and the synthesized speech of the macroscopic path guidance information are mixed in a dual-channel manner and sent to the mechanical vibrator of the bone conduction headphones. The audio is transmitted to the auditory nerve through the bones, and the physical vibration conversion and superposition of these two types of digital audio signals are performed. The system clock of the processing terminal is synchronized with the network time protocol of the smart terminal at the microsecond level. The playback timestamp of the buffer is set to ensure that the two audio segments trigger the digital-to-analog converter at the same time scale, thereby completing the merging, parsing and synchronous output of voice prompts and navigation guidance information on the hardware side.

Claims

1. An obstacle avoidance navigation system based on a camera and a ToF module, characterized in that, The system includes: The camera module is used to perform the steps of acquiring scene images and outputting the scene images; The ToF module is used to perform the steps of acquiring a scene depth map and outputting the scene depth map. The target recognition module is used to perform the step of detecting key targets in the scene image using a convolutional neural network; The data fusion module is used to perform the step of mapping the key target and the scene depth map to a spatial grid to obtain the grid mapping result; The obstacle determination module is used to perform the step of determining obstacles based on the grid mapping results and obtaining the determination results; The navigation prompt module is used to perform the steps of generating a navigation table based on the determination result and outputting voice prompts; The communication coordination module is used to perform the steps of establishing a communication connection with the smart terminal and synchronously outputting the voice prompts. 2.The camera and ToF module based obstacle avoidance navigation system of claim 1, wherein, The specific steps for acquiring and outputting scene images are as follows: The system is equipped with a wide-angle camera integrated into the smart glasses. The wide-angle camera is used to capture scene images. The frame rate of the wide-angle camera is adjusted to a preset frame rate. The captured scene images are compressed in JPEG format. The compressed scene images are then transmitted to a processing terminal and output as scene images.

3. The obstacle avoidance navigation system based on a camera and a ToF module according to claim 2, characterized in that, The specific steps for obtaining and outputting the scene depth map are as follows: A planar array sensor is configured and installed parallel to the optical axis of the wide-angle camera. The resolution of the planar array sensor is set to a preset resolution. The planar array sensor is used to acquire a scene depth map. The acquisition frame rate of the planar array sensor is controlled to be synchronized with the frame rate of the wide-angle camera. The scene depth map is filtered. The filtered scene depth map is then aligned with coordinates and output.

4. The obstacle avoidance navigation system based on a camera and a ToF module according to claim 2, characterized in that, The specific steps for detecting key targets in the scene image using a convolutional neural network are as follows: A convolutional neural network is deployed in the processing terminal. The scene image is input into the convolutional neural network, and the semantic features of the scene image are extracted using the convolutional neural network. Multiple preset categories of targets in the semantic features are identified, and the multiple preset categories of targets are summarized to obtain key targets. The preset categories of targets include obstacle targets and road guidance targets.

5. The obstacle avoidance navigation system based on a camera and a ToF module according to claim 1, characterized in that, The specific steps for mapping the key target and the scene depth map to a spatial grid to obtain the grid mapping result are as follows: The field of view is divided into an M x M spatial grid. Each grid in the spatial grid corresponds to a preset physical size. The position coordinates of the key target are extracted and mapped to the corresponding grid position in the spatial grid. A visual recognition state label is assigned to the corresponding grid position to obtain the visual recognition state corresponding to the spatial grid. The depth point cloud data of the scene depth map is extracted and mapped to the corresponding grid position in the spatial grid. A depth detection state label is assigned to the corresponding grid position to obtain the depth detection state corresponding to the spatial grid. The visual recognition state and the depth detection state in the spatial grid are integrated to obtain the grid mapping result.

6. The obstacle avoidance navigation system based on a camera and a ToF module according to claim 5, characterized in that, The specific steps for determining obstacles based on the mesh mapping results and obtaining the determination results are as follows: Extract the visual recognition state and the depth detection state of each grid in the grid mapping result, and determine the visual recognition state and the depth detection state; If the visual recognition state is an obstacle state and the depth detection state is an obstacle state, the grid is confirmed to be impassable, and a judgment result is obtained. If the visual recognition state is an obstacle state and the depth detection state is a non-obstacle state, extract the depth distance data corresponding to the depth detection state, and determine whether the depth distance data corresponding to the depth detection state is less than or equal to a preset distance threshold. If the depth distance data corresponding to the depth detection state is less than or equal to the preset distance threshold, confirm that the grid is in an avoidance state. If the depth distance data corresponding to the depth detection state is greater than the preset distance threshold, confirm that the grid is in a pending confirmation state, and obtain the determination result. If the visual recognition state is a non-obstacle state and the depth detection state is an obstacle state, extract the depth distance data corresponding to the depth detection state, and determine whether the depth distance data corresponding to the depth detection state is less than or equal to the preset distance threshold. If the depth distance data corresponding to the depth detection state is less than or equal to the preset distance threshold, confirm that the grid is in an avoidance state. If the depth distance data corresponding to the depth detection state is greater than the preset distance threshold, confirm that the grid is in a pending confirmation state, and obtain the determination result. If the visual recognition state is a non-obstacle state and the depth detection state is a non-obstacle state, the grid is confirmed to be passable, and a judgment result is obtained.

7. The obstacle avoidance navigation system based on a camera and a ToF module according to claim 6, characterized in that, The specific steps for generating a navigation table and outputting voice prompts based on the determination result are as follows: Extract the impassable state, avoidance state, and passable state from the determination result. Generate a navigation table of preset dimensions based on the impassable state, avoidance state, and passable state. Locate the user's current position in the navigation table. Expand to the left by a preset number of steps from the current position in the navigation table. Calculate the number of consecutive passable grids corresponding to the left side. Expand to the right by the preset number of steps from the current position in the navigation table. Calculate the number of consecutive passable grids corresponding to the right side. Compare the number of consecutive passable grids corresponding to the left side and the number of consecutive passable grids corresponding to the right side. Filter the direction corresponding to the maximum value of the consecutive passable grids. Generate one of the left-side passable prompt sound and the right-side passable prompt sound based on the direction corresponding to the maximum value, and output the voice prompt.

8. The obstacle avoidance navigation system based on a camera and a ToF module according to claim 2, characterized in that, The specific steps for establishing a communication connection with the smart terminal and synchronously outputting the voice prompts are as follows: A Bluetooth communication connection is established between the processing terminal and the smart terminal. The voice prompt is sent to the navigation application installed inside the smart terminal. The bone conduction headphones are invoked to play the voice prompt received by the navigation application. The voice prompt is output synchronously in conjunction with the path guidance information of the navigation application.

9. The method of the obstacle avoidance navigation system based on a camera and a ToF module according to any one of claims 1-8, characterized in that, Includes the following steps: Acquire scene images and output the scene images; Obtain the scene depth map and output the scene depth map; The key targets are obtained by detecting the scene image using a convolutional neural network; The key targets are mapped to the scene depth map onto a spatial grid to obtain the grid mapping result; Obstacles are determined based on the grid mapping results, and the determination results are obtained. A navigation table is generated based on the determination result, and voice prompts are output. Establish a communication connection with the smart terminal and output the voice prompts synchronously.

Images