FPGA-based intelligent guiding method and helmet system

Abstract

The application provides an intelligent blind guiding method and a helmet system based on FPGA, and steps are as follows: calling a blind navigation map and a GPS / Beidou navigation module, planning a walking route; a binocular camera collects images in real time; a FPGA processing module performs target detection on the input images through a multi-target detection algorithm, performs instance segmentation on road targets, and performs binocular distance measurement through a binocular distance measurement algorithm; the FPGA processing module performs hardware acceleration on the multi-target detection algorithm; accurate distance measurement of close-range obstacles is performed; navigation information, multi-target detection results, binocular distance measurement results and ultrasonic distance measurement results are fused, decision algorithm analysis is performed, and navigation voice prompts are given; when an image picture changes slightly, binocular and target detection are not performed. The application accelerates multi-target detection and binocular distance measurement based on FPGA, reduces power consumption while ensuring detection accuracy and real-time performance.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more particularly to an FPGA-based intelligent guidance method for the blind and a helmet-mounted system. Background Technology

[0002] With the improvement of people's living standards, blind people have a strong demand for improved quality of life, which in turn drives up the market demand for intelligent visual aids. However, the production technology of intelligent visual aids for the blind is highly complex, and the products are expensive, thus they have long been neglected. Currently, the assistive devices carried by blind people are mainly guide canes and guide glasses. However, due to their shape and usage characteristics, they cannot provide a comprehensive view of the surrounding environment. Furthermore, current guide devices primarily focus on obstacle avoidance and cannot adequately meet the navigation needs of blind people.

[0003] In recent years, computer vision has developed rapidly with the support of machine learning and deep learning. Despite significant progress and the powerful capabilities of computer vision, object detection remains a complex process fraught with challenges. One challenge is the diversity of viewpoints; the same object appears completely different from different angles, thus the detector's goal is to identify objects from various perspectives. Secondly, deformation and occlusion make it difficult to detect the true category and distance of objects, further complicating target identification and threatening accuracy. Simultaneously, in video processing, detectors need to be trained to analyze constantly changing environments. This means that object detection algorithms must not only accurately classify important targets but also have high prediction speeds to recognize constantly changing targets. Ensuring accuracy and real-time performance requires powerful computing capabilities, but GPU-based algorithm acceleration hardware consumes significant power, making it difficult to meet the practical needs of navigation for the blind. Existing guide devices for the blind rarely offer comprehensive route navigation functions, and their prices are high.

[0004] One invention patent discloses a smart navigation method for the blind using multi-view stereo vision. It employs a multi-view image acquisition device to collect image data, which is then used to obtain camera calibration results and acquire positioning information using a pre-set calibration method. The image data, positioning information, and camera calibration results are uploaded to a pre-set data processing center. A 3D model is obtained, and based on the image data (red / green indicator), positioning information, and camera calibration results, obstacle detection, moving object tracking, face detection, and traffic light detection are performed to plan the blind person's route. The blind person's route is further planned based on traffic light prompts, person recognition data, fixed obstacle positions, and moving obstacle trajectories. Based on these factors, a warning signal is generated and issued to provide sensory alerts for the blind. This invention solves the technical problems of large guidance errors, poor portability, and high operating costs. However, the aforementioned invention patent only uses a camera and requires data to be uploaded to a remote guidance platform for processing, resulting in low obstacle ranging accuracy and large latency. Summary of the Invention

[0005] To address the technical problems of high power consumption and high cost of existing guide methods for the blind, this invention proposes an intelligent guide method and helmet-mounted system based on FPGA. This system solves the problem of excessive power consumption caused by high-precision and high-computing power of multi-target detection at a lower cost, while providing functions such as voice route navigation and obstacle avoidance reminders for blind people.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows: an intelligent guidance method for the blind based on FPGA, comprising the following steps:

[0007] S1: The user calls the built-in blind navigation map and GPS / BeiDou navigation module in the helmet through the voice recognition module and voice prompt module, and sets the destination and departure point information; the blind navigation map plans the walking route according to the characteristics of the blind person and provides feedback to the user. After the user confirms, the navigation function is activated.

[0008] S2: The binocular camera captures images from the front and back of the helmet in real time and transmits the images to the FPGA processing module;

[0009] S3: The FPGA processing module performs multi-target detection algorithm on the input image to detect targets and performs binocular ranging algorithm to achieve binocular ranging.

[0010] S4: The FPGA processing module performs hardware acceleration of the multi-target detection algorithm, and the acceleration work adopts a phased convolution operation acceleration method.

[0011] S5: Uses an ultrasonic ranging module to accurately measure the distance to nearby obstacles;

[0012] S6: Integrates navigation information, multi-target detection results, binocular ranging results, and ultrasonic ranging results, and provides detailed navigation voice prompts for the blind after analysis by a decision algorithm;

[0013] S7: Adopt an adaptive energy-saving strategy and analyze the dynamic degree of the environment in all directions through the shadow difference method. When the image changes slightly, the multi-target detection and binocular ranging in step S3 will not be performed.

[0014] Preferably, the multi-target detection algorithm detects common objects in traffic scenes, and the binocular ranging algorithm calculates the distance between the detected target objects and the camera. The specific implementation method is as follows:

[0015] (1) The input image is processed by YOLO V5 neural network inference and nonmaximum suppression to generate target boxes;

[0016] (2) The binocular ranging algorithm processes the input image to generate a depth map, and calculates the distance between each target and the camera based on the coordinates of the depth map and the target box;

[0017] (3) The road image within the target box is semantically segmented using YOLO V5 to obtain the precise location of the road;

[0018] (4) Analyze the information of the identified traffic lights and inform the current road traffic conditions.

[0019] Preferably, the multi-target detection algorithm adopts the YOLO V5 algorithm and utilizes the FPGA processing module for hardware acceleration of convolution operations based on the Winograd algorithm; the tensor dimension transformation and nonlinear activation function operation of the output header of the YOLO V5 algorithm are processed on the CPU, while the convolution operations of other parts of the neural network are accelerated by the FPGA hardware; wherein, the convolution operations of other parts of the neural network are accelerated by an improved convolution operation hardware acceleration unit designed based on the Winograd algorithm idea.

[0020] Preferably, the overall process of hardware acceleration of convolution operations based on the Winograd algorithm in the FPGA processing module is as follows: the FPGA processing module collects real-time information of the surrounding environment through two binocular cameras and stores the data in the memory; the relevant information of the neural network weights of the YOLO V5 algorithm for multi-object detection is stored in the memory in advance, and the Winograd transform convolution kernel is directly stored in the memory. The memory is configured with feature map cache and weight cache to store the input image and the Winograd transform convolution kernel, respectively; the programmable terminal inside the chip implements hardware acceleration of the convolution unit and pooling unit, and completes data transmission through DMA; after the hardware acceleration of the convolution layer and pooling layer of the neural network, the result is output to the CPU in the FPGA chip.

[0021] Preferably, the hardware acceleration method for convolution operations based on the Winograd algorithm is as follows: For an image with an input size of H×W, the image is divided into several image blocks, each with a size of h×w, and the output parameter size is m×n. The convolution operation with a kernel size of r×r is as follows:

[0022] Y = A T [GgG] T ]⊙[B T dB]]A

[0023] Where ⊙ represents the dot product of two matrices, i.e., multiplying corresponding elements; g represents a convolution kernel of size r×r; d represents an image patch of size (m+r-1)×(m+r-1); and G represents the kernel transformation matrix of size (m+r-1)×r, which is the Winograd transformation convolution kernel GgG. T B T A represents the input transformation matrix with size (m+r-1)×(m+r-1); T The output transformation matrix has a size of m×(m+r-1); where h=m+r-1, w=n+r-1; Y represents the result of the convolution operation; finally, the results obtained from all image blocks are stitched together.

[0024] Preferably, the memory stores Winograd transform convolution kernels, wherein the convolution operation between the two-dimensional feature map and the two-dimensional convolution kernel on the same channel in the memory is as follows:

[0025] The two-dimensional feature map d stored in the same channel of memory and the Winograd transform convolution kernel are respectively fed into two registers of a convolution operation acceleration unit via the AXI bus; then, the two-dimensional feature map d undergoes B transformation through the input feature map transformation module. T The dB operation yields the transformed feature map. Then, the transformed feature map and the Winograd transform convolution kernel are simultaneously fed into the matrix dot product operation unit to perform the ⊙ operation, outputting the result R. Finally, the result R is processed by the output feature map transformation module to obtain the result Y. The result Y is returned to the register and written back to the memory via the AXI bus.

[0026] For accelerated convolution operations on a set of 3D feature maps and a 3D convolution kernel, a set of 3D feature maps and a 3D convolution kernel are retrieved from the feature map cache and weight cache, respectively. In the calculation of each channel of the set of 3D feature maps and the 3D convolution kernel, a convolution operation acceleration unit is used to complete the calculation in parallel, resulting in multiple 2D feature maps. An accumulation module adds the values ​​at the same positions of the multiple 2D feature maps to obtain a single feature map. Multiple sets of convolution operation acceleration units complete the calculation of a set of feature maps and a 4D convolution kernel in parallel, resulting in a 3D feature map as the output. The above process is performed on each set of input feature maps, and the final output results constitute a complete output feature map. The circuits of the input feature map transformation module and the output feature map transformation module both adopt the idea of ​​merging like terms in matrix operation results. For addition or subtraction operations of the same variables in a polynomial, a single adder or subtractor is used to implement matrix multiplication operations.

[0027] Preferably, the decision algorithm comprises the following steps:

[0028] 1) Estimating target distance: Set the maximum ultrasonic ranging distance to M meters. When the ultrasonic ranging result is the maximum distance, the distances of each target measured by the binocular ranging algorithm are taken as the standard. When the ultrasonic ranging result is less than the maximum distance, it is determined that an obstacle exists. Then, the target closest to the ultrasonic ranging result is found in the binocular ranging result. This target is identified as the current obstacle, and its distance is D. If the target cannot be found in the binocular ranging result, the current obstacle is considered to be an unknown obstacle, and the ultrasonic ranging result is D.

[0029] 2) Computational environment complexity:

[0030] Where H represents the complexity of the environment, and N represents the number of targets. Indicates the rate of change of the target. This indicates the impact of target distance on the results; a1, a2, and a3 are the corresponding impact coefficients.

[0031] 3) Determine whether to send a notification to the user based on the complexity of the environment: when the environmental complexity H is greater than or equal to the set value H t Send a reminder command if the message is not sent; otherwise, do not send a reminder command.

[0032] 4) Determine the reminder type, and select N, and The one with the greatest impact is used as the alert type; when it is judged as N, the user is alerted that the surrounding environment is chaotic; when it is judged as When the environment is changing, it alerts the user; when it is determined to be... When the wearer is too close to surrounding targets, the device will alert the outside world to keep their distance from the user and activate the flashlight.

[0033] Preferably, when the user is at a large intersection, the system detects the traffic flow, pedestrian flow, and traffic light changes during that time period, and provides real-time prompts to the user to proceed according to traffic rules; if the user deviates from the normal pedestrian crossing or route, the system prompts the user to correct their course; when the user is on a tactile paving obstructed by obstacles, the system detects the position and size of the obstacles on the tactile paving and provides the information to the user via voice, allowing them to avoid them in advance or change their course; if an object approaches the user at high speed, a flashing light will sound an alarm, and a voice prompt will prompt the user to take appropriate evasive action.

[0034] A smart helmet system based on an FPGA-based intelligent guidance method for the blind includes a helmet body. The helmet body is equipped with an FPGA processing module, an ultrasonic ranging module, a binocular camera, a voice recognition module, a voice prompt module, a Bluetooth module, a flexible solar panel, a flashing warning module, a lithium battery and power management module, and a GPS / BeiDou navigation module. All these modules are mounted on the helmet body. The flexible solar panel is connected to the lithium battery and power management module. The ultrasonic ranging module, binocular camera, voice recognition module, voice prompt module, Bluetooth module, flashing warning module, lithium battery and power management module, and GPS / BeiDou navigation module are all connected to the FPGA processing module. The Bluetooth module is connected to a mobile terminal.

[0035] Preferably, the flexible solar panel covers the upper surface of the helmet body, and the GPS / BeiDou navigation module is positioned directly above the helmet body for real-time positioning of the helmet body; the flashing warning module includes a flashlight located on the rear side of the helmet body to alert pedestrians and oncoming vehicles; wide-angle binocular cameras are mounted at both the front and rear of the helmet body, and ultrasonic ranging modules are mounted at the front and sides of the helmet body; the voice recognition module is connected to a microphone located on the lowermost inner wall of the side of the helmet body; the voice prompt module broadcasts voice prompts through a speaker.

[0036] The FPGA processing module includes an FPGA chip with a built-in CPU core, a memory, a power management circuit, and an interface circuit. The FPGA chip is connected to the memory and the power management circuit, and the FPGA chip is connected to an ultrasonic ranging module, a binocular camera, a voice recognition module, a voice prompt module, a Bluetooth module, a flashing warning module, a lithium battery and power management module, and a GPS / BeiDou navigation module through the interface circuit. The lithium battery and power management module includes at least a lithium battery and a multi-channel power management module, and the lithium battery and power management module is connected to a flexible solar panel.

[0037] The FPGA chip is equipped with a multi-target detection unit and a binocular ranging unit to realize multi-target detection of images and target distance detection. The programmable end of the FPGA chip is equipped with a convolution operation acceleration unit for hardware acceleration.

[0038] The sampling rate of the binocular camera is adaptively adjusted according to the environment. When the number of targets is stable and the target distance is safe, the FPGA processing module checks once every few frames.

[0039] The beneficial effects of this invention are:

[0040] (1) The helmet system of the present invention includes a voice recognition and voice prompt module, and has a built-in map and Beidou / GPS positioning system, so that blind people can safely reach their destination by following the voice navigation throughout the journey after traveling to the designated destination.

[0041] (2) In normal operation mode, the present invention adopts a multi-target detection algorithm that embeds binocular ranging and road instance segmentation algorithm, and combines ultrasonic ranging to make the detection results more accurate.

[0042] (3) This invention utilizes the high energy efficiency of FPGA to accelerate multi-target detection algorithm and binocular ranging algorithm based on FPGA, thereby reducing power consumption while ensuring detection accuracy and real-time performance; and effectively reminds the wearer of nearby obstacles through real-time voice broadcast.

[0043] (4) The helmet system of the present invention can protect the wearer’s head and neck to the maximum extent in the event of an accident, enabling blind people to travel safely in their daily lives.

[0044] (5) The helmet-type system of the present invention, combined with the characteristics of flexible solar panels, can charge the system in real time and improve the system's endurance.

[0045] (6) The helmet system of the present invention has low cost, reduces the economic burden on users, improves the safety of blind people traveling, and has high economic benefits. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the helmet-type system of the present invention.

[0048] Figure 2 for Figure 1 The right view shown.

[0049] Figure 3 This is a flowchart of the binocular ranging algorithm of the present invention.

[0050] Figure 4 This is a flowchart of the YOLO V5 algorithm and binocular ranging algorithm of the present invention.

[0051] Figure 5 This is a flowchart of the guide method for the blind according to the present invention.

[0052] Figure 6 This is a schematic diagram illustrating the process of accelerating the convolution operation in an embodiment of the present invention.

[0053] Figure 7 This is a schematic diagram of the convolution operation acceleration unit according to an embodiment of the present invention.

[0054] Figure 8 This is a schematic diagram of the circuit structure of the helmet-type system of the present invention.

[0055] Figure 9 This is a schematic diagram of the circuit operation of the input feature map transformation module in an embodiment of the present invention.

[0056] Figure 10 This is a circuit operation diagram of the output feature map transformation module in an embodiment of the present invention.

[0057] In the diagram, 1-voice prompt module, 2-voice recognition module, 3-GPS / BeiDou navigation module, 4-flexible solar panel, 5-lithium battery and power management module, 6-Bluetooth module, 7-ultrasonic ranging module, 8-binocular camera; 9-flash warning module, 10-FPGA processing module. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Example 1

[0060] like Figure 1 and Figure 2As shown, this embodiment of the invention provides an FPGA-based smart helmet system, including a helmet body, an FPGA processing module 10, an ultrasonic ranging module 7, a binocular camera 8, a voice recognition module 2, a voice prompt module 1, a Bluetooth module 6, a flexible solar panel 4, a flashing warning module 9, a lithium battery and power management module 5, and a GPS / BeiDou navigation module 3. The FPGA processing module 10, ultrasonic ranging module 7, binocular camera 8, voice recognition module 2, voice prompt module 1, Bluetooth module 6, flexible solar panel 4, flashing warning module 9, lithium battery and power management module 5, and GPS / BeiDou navigation module 3 are all mounted on the helmet body. The flexible solar panel 4 is connected to the lithium battery and power management module 5. The ultrasonic ranging module 7, binocular camera 8, voice recognition module 2, voice prompt module 1, Bluetooth module 6, flashing warning module 9, lithium battery and power management module 5, and GPS / BeiDou navigation module 3 are all connected to the FPGA processing module 10. The Bluetooth module 6 is connected to a mobile terminal.

[0061] Figure 1 For illustrative purposes only, a flexible solar panel 4 covers the upper surface of the helmet body to collect solar energy during use. A lithium battery and power management module 5 controls the charging of the lithium battery and supplies power to all modules. A GPS / BeiDou navigation module 3 is positioned directly above the helmet body for real-time positioning. Apart from these two, all other modules are mounted on the sides of the helmet. A flashing warning module 9, including a flashlight, is located on the rear of the helmet body to alert pedestrians and oncoming vehicles. Wide-angle binocular cameras are installed at both the front and rear of the helmet body to capture image information from the front or rear, ensuring multi-directional image detection. Ultrasonic ranging modules 7 are installed at the front and sides of the helmet body to capture multi-directional distance information and reduce blind spots. A voice recognition module 2, a voice broadcasting module 1, and a Bluetooth module 6 are located above the ultrasonic ranging modules on the left and right sides of the helmet body. The voice recognition module 2 is connected to a microphone for recognizing voice information; the voice prompt module 1 broadcasts voice prompts through a speaker. The microphone is located on the innermost side of the helmet body, at the bottom, for collecting voice information. The helmet system adaptively adjusts the sampling rate of the binocular cameras according to the environment. When the number of targets is stable and the target distance is safe, target detection can be paused. Target detection will resume after the number of frames skipped during the pause reaches a preset value. The Bluetooth module has Bluetooth communication capabilities and can connect to Bluetooth devices such as mobile phones to achieve remote communication or alarm / help functions.

[0062] The FPGA processing module 10 includes an FPGA chip with a built-in CPU core, memory, power management circuit, USB interface, serial port interface, IIC interface, SPI interface, and GPIO interface. The USB interface supports system charging and system software upgrades. The lithium battery and power management module 5 includes at least a high-performance lithium battery and a power management module to power the various modules in the system. The power management module is connected to the power management circuit. The memory stores the acquired image information. The FPGA chip is connected to both the memory and the power management circuit. The FPGA chip is connected to the ultrasonic ranging module 7, binocular camera 8, voice recognition module 2, voice prompt module 1, Bluetooth module 6, flashing warning module 9, lithium battery and power management module 5, and GPS / BeiDou navigation module 3 via interface circuits. The FPGA chip is also connected to the USB interface, serial port interface, IIC interface, SPI interface, and general-purpose GPIO interface. The serial port interface, IIC interface, SPI interface, and GPIO interface control and communicate with other modules. These multiple interfaces facilitate compatibility with various products.

[0063] The FPGA chip is equipped with a multi-target detection unit and a binocular ranging unit to realize multi-target detection of images and target and distance detection. The programmable end of the FPGA chip is equipped with a convolution operation acceleration unit.

[0064] like Figure 8 As shown, the flexible solar panel 4 is connected to the lithium battery and power management module 5, which supplies power to all modules of the helmet system. The FPGA processing module 10 is connected to the binocular camera and accelerates image processing through multi-target detection and binocular ranging algorithms. The FPGA processing module 10 receives instructions recognized by the voice recognition module 2 and calls the built-in map and GPS / BeiDou navigation module 3 in the embedded custom operating system to realize navigation functions. The FPGA processing module 10 sends information according to navigation needs and selects to provide voice and light reminders to the helmet user and other traffic participants. The embedded custom operating system is connected to the user's smartphone through the Bluetooth module 6 and, in conjunction with the microphone in the voice recognition module 2 and the speaker in the voice prompt module 1, realizes functions such as making calls.

[0065] In addition to the five essential functions of CPU management, memory management, device management, file management, and job management, the embedded custom operating system also includes a built-in map system for navigation for the blind and supports lane-level navigation of the Beidou navigation system.

[0066] Furthermore, in this embodiment of the invention, the helmet system may also include a bone conduction module to provide bone conduction voice prompts; in case of an emergency, the wearer can still hear ambient sounds while receiving bone conduction voice prompts, thus avoiding the danger of accidents caused by not being able to hear external sounds.

[0067] In embodiments of the present invention, two operating modes may be included: intelligent power-saving mode and forced power-saving mode. In intelligent power-saving mode, all modules in the helmet-mounted system are in operation, employing an adaptive power-saving strategy that adjusts the sampling rate of the binocular camera, i.e., adjusts the image processing rate of the PL terminal. In forced power-saving mode, the binocular camera and GPS / BeiDou navigation are in sleep mode, while other modules in the helmet-mounted system are in operation. When the wearer is in motion relative to surrounding objects, the camera is on and performing normal detection; the system does not automatically decide to turn off the camera, and the helmet-mounted system is in intelligent power-saving mode. The helmet-mounted system defaults to intelligent power-saving mode; when the battery is low, the system automatically switches to forced power-saving mode, which the user can actively switch via voice.

[0068] Example 2

[0069] To address the shortcomings of existing technologies, this invention provides a specific implementation method for an FPGA-based intelligent guidance method for the visually impaired, such as... Figure 5 As shown, the specific steps include:

[0070] S1: The user calls up the map built into the helmet for navigation for the blind through the voice recognition module and voice prompt module, and sets the destination and departure point information; the system plans the walking route according to the characteristics of the blind and provides feedback to the wearer. After the wearer confirms, the navigation function is activated.

[0071] Specifically, when a user specifies a destination by voice, the embedded custom operating system receives the voice recognition command and calls the GPS / BeiDou navigation module and voice prompt module to provide real-time voice broadcast of distance and route.

[0072] S2: The binocular camera captures images from the front and rear in real time and transmits the image information to the FPGA processing module.

[0073] Specifically, the helmet-mounted system has two binocular cameras installed at the front and rear of the helmet body to comprehensively capture the wearer's surrounding environment. The binocular cameras transmit the real-time image information to the connected FPGA processing module, which performs multi-target detection and binocular ranging on the images. At the same time, the sampling rate is adaptively adjusted according to the environment. When the number of targets is stable and the target distance is appropriate, target detection can be paused. After the number of frames skipped during the pause reaches a preset value, target detection resumes.

[0074] S3: The FPGA processing module executes a multi-target detection algorithm for target detection and executes a binocular ranging algorithm for binocular ranging.

[0075] like Figure 4As shown, the multi-object detection algorithm will detect common objects in traffic scenes, and the binocular ranging algorithm will calculate the distance between the detected target objects and the camera.

[0076] Specifically, the multi-target detection algorithm and binocular ranging algorithm in this embodiment have the following characteristics:

[0077] (1) The input image is processed by YOLO V5 neural network inference and nonmaximum suppression to generate target boxes.

[0078] (2) The binocular ranging algorithm processes the input image to generate a depth map, and calculates the distance between each target and the camera based on the coordinates of the depth map and the target box.

[0079] (3) Perform instance segmentation on the road image using YOLO V5, and calculate the precise location of the road based on the identified target box position.

[0080] (4) Analyze the information of the identified traffic lights and inform the current road traffic conditions.

[0081] S4: The FPGA processing module performs hardware acceleration for the multi-target detection algorithm, and the acceleration process is carried out in stages by enabling the convolution operation acceleration method.

[0082] A complete convolutional neural network for multi-object detection has the following characteristics:

[0083] ① The initial part of the network may use modules such as Focus that have frequent slicing operations but low computational load;

[0084] ② The network output header typically undergoes numerous tensor dimension transformations and non-linear activation functions such as Sigmoid;

[0085] ③ The main part consists of convolution operations. During the forward inference process of the network, the image is downsampled multiple times, resulting in a trend of smaller size.

[0086] The segmentation strategy is as follows: stages ① and ② are processed on the CPU, while stage ③ is accelerated using FPGA hardware. For the later stages of stage ③, a hardware acceleration unit for convolution operations, designed based on the Winograd algorithm, is used for accelerated processing.

[0087] Specifically, the multi-object detection algorithm adopts the YOLO V5 algorithm and utilizes FPGA-based convolution operation hardware acceleration for processing, with the acceleration method as follows:

[0088] The algorithm is accelerated using FPGA hardware in the FPGA processing module. The overall process is as follows: The FPGA processing module acquires real-time information about the surrounding environment through two binocular cameras (front and rear) and stores the data in memory, such as DDR3. The neural network weights and other relevant information of the YOLO V5 multi-object detection algorithm are pre-stored in memory. The Winograd transform convolution kernel, i.e., GgG... T The data is also pre-stored in memory; after the system starts, initialization settings are completed; the programmable logic controller (PL) inside the FPGA chip implements IP hardware acceleration for convolutional and pooling units, and data transfer is completed via DMA, which reduces the workload of the CPU. After hardware acceleration of the convolutional and pooling layers of the neural network, the results are output to the CPU inside the FPGA; after subsequent result extraction and visualization, the results are recorded in the device's built-in storage in the form of log files, images, videos, etc., to complete the detection of the target.

[0089] For image data with input size H×W, the image data is divided into several image blocks, each with size h×w. Then, for each group, a convolution operation is performed with output parameter size m×n and convolution kernel size r×r according to the following formula:

[0090] Y = A T [GgG] T ]⊙[B T dB]]A

[0091] Where ⊙ represents the multiplication of corresponding elements of two matrices; g represents a convolution kernel of size r×r; d represents an image patch of size (m+r-1)×(m+r-1); G represents the kernel transformation matrix of size (m+r-1)×r; B T A represents the input transformation matrix with size (m+r-1)×(m+r-1); T The output transformation matrix has a size of m×(m+r-1); where h=m+r-1, w=n+r-1; Y represents the result of the convolution operation, and finally the results obtained from all image blocks are stacked.

[0092] In this embodiment, the relevant parameters adopt the following specific values:

[0093]

[0094] In the specific implementation, m=2, n=2, r=3, h=4, w=4 are selected.

[0095] In the process of accelerating convolution operations using a set of 3D feature maps and a 3D convolution kernel, it is necessary to pass through a convolution operation acceleration unit, such as... Figure 7As shown, firstly, the two-dimensional feature map d and the two-dimensional convolution kernel g, stored on the same channel in the DDR3 memory, are respectively entered into two registers of a convolution operation acceleration unit via the AXI bus; then, the two-dimensional feature map d undergoes B transformation through the input feature map transformation module. T The dB operation yields the transformed feature map. Then, the transformed feature map and the Winograd transform convolution kernel are simultaneously fed into the matrix dot product operation unit to perform the ⊙ operation and output the result R. Finally, the result R is processed by the output feature map transformation module to obtain the result Y. The result Y is returned to the register and written back to the memory DDR3 via the AXI bus.

[0096] The above process involves the convolution operation of a two-dimensional feature map and a two-dimensional convolution kernel on the same channel in DDR3 memory. For accelerated convolution operations using a set of three-dimensional feature maps and a three-dimensional convolution kernel, please refer to [link to documentation]. Figure 6 , Figure 6 This is a schematic diagram of the convolution acceleration process. (Example) Figure 6 As shown, a set of 3D feature maps and a 3D convolutional kernel are taken from the feature map cache and weight cache, respectively. In the calculation of each channel of the set of 3D feature maps and the 3D convolutional kernel, a convolution operation acceleration unit is used to complete the calculation in parallel, resulting in multiple 2D feature maps as output results. The above output results are passed through the accumulation module to sum the values ​​at the same position of the multiple 2D feature maps to obtain a feature map. Multiple sets of convolution operation acceleration units complete the calculation of a set of feature maps and a 4D convolutional kernel in parallel to obtain a 3D feature map as output results. The above process is completed for each set of input feature maps, and the final output results constitute a complete output feature map.

[0097] Specifically, please refer to the circuit implementation of the input feature map transformation module. Figure 9 , Figure 9 This is a schematic diagram of the circuit structure of the input feature map transformation module. In the input feature map transformation module of this embodiment, the idea of ​​merging like terms in the matrix operation results is adopted. That is, for addition or subtraction operations with the same variables in the polynomial, a single adder or subtractor is reused to achieve the transformation of B. T The dB operation is simplified from the original 48 adders and subtractors to 32 adders and subtractors, as shown in the following implementation:

[0098] Input image B T The original calculation result of dB is

[0099] In the input feature map transformation module, elements such as d8-d that appear multiple times in the matrix are processed. 10 d9+d 10 Operations involving equal variables are performed multiple times using only one arithmetic unit, resulting in a combined B. T The dB matrix results are as follows:

[0100] Where, d 8s10 Indicates d8-d 10 d 9a10 It means d9+d 10 Other expressions follow the same pattern; ultimately, only 32 adders and subtractors are needed to process B. T Operations on the dB matrix.

[0101] Similarly, please refer to the circuit implementation of the output feature map transformation module. Figure 10 , Figure 10 This is a schematic diagram of the circuit structure for the output feature map transformation module. Figure 10 The hardware circuit operation process is represented. Indicates an adder. This represents a subtractor. Both adders and subtractors have two input terminals on the left and an output terminal on the right.

[0102] S5: Uses an ultrasonic ranging module to accurately measure the distance to nearby obstacles.

[0103] Specifically, the ultrasonic ranging module is installed on the front and left and right sides of the helmet to avoid the inaccurate measurement caused by the visual recognition module in dim ambient light, and to serve as an auxiliary ranging function; the ultrasonic ranging range is 0.1-3 meters, and is more accurate.

[0104] S6: By integrating navigation information, multi-target detection results, binocular ranging results, and ultrasonic ranging results, and after analysis by a decision algorithm, it provides detailed voice prompts for navigation to the blind.

[0105] Specifically, the steps of the decision-making algorithm are as follows:

[0106] 1) Estimating Target Distance: The maximum ultrasonic ranging distance is set to M = 3 meters. When the ultrasonic ranging result is the maximum distance, the distances to targets measured by the binocular ranging algorithm are used. When the ultrasonic ranging result is less than the maximum distance, an obstacle is considered to exist. The binocular ranging results are then used to find the target closest to the ultrasonic ranging result; this target is identified as the current obstacle, and its distance is assigned as D. If no target can be found in the binocular ranging results, the current obstacle is considered an unknown obstacle, and the ultrasonic ranging result is assigned as D.

[0107] 2) Computational environment complexity:

[0108]

[0109] Where H represents the complexity of the environment, and N represents the number of targets (N can be calculated based on the target detection results; if the ultrasonic result is not the maximum value, it is also counted as one target). Indicates the rate of change of the target. This indicates the impact of target distance on the result; a1, a2, and a3 are the corresponding impact coefficients.

[0110] 3) Whether to send a reminder to the wearer based on the complexity of the environment: when the environmental complexity H is greater than or equal to the set value H t Send a reminder command; otherwise, do not send one.

[0111] 4) Determine the reminder type, and select N, and The factor with the greatest impact is used as the alert type (based on numerical comparison); when the value is N, the wearer is alerted that the surrounding environment is chaotic; when the value is N... At times, it reminds the wearer that the surrounding environment is constantly changing; when judged to be When the wearer is too close to surrounding targets, the flashlight will alert others to keep their distance.

[0112] S7: Employs an adaptive energy-saving strategy, primarily using the difference image method to analyze the dynamic range of the environment in all directions. When the image changes minimally, binocular detection and target detection will not be performed. Similarly, the ultrasonic ranging module adjusts the detection frame rate based on the rate of change of the measured distance. Furthermore, it will assign different levels of attention to each camera based on the user's specific situation and the actual environment, controlling their capture frame rate accordingly.

[0113] Furthermore, when the user is at a large intersection, the helmet-mounted system will detect the traffic flow, pedestrian flow, and traffic light changes during that time period (by detecting targets, the system records the number of targets within that time period) and provide real-time prompts to the user to proceed according to traffic rules; if the user deviates from the normal pedestrian crossing or route, the system will prompt the user to correct their route; when the user is on a tactile paving obstructed by obstacles, the helmet-mounted system will detect the position and size of the obstacles on the tactile paving (the ratio of the target area to the size of the entire image) and provide this information to the user in advance via voice, allowing them to avoid the obstacle or change their route; if an object approaches the user at high speed (the same target, but with varying sizes over a certain period of time), the helmet-mounted system will issue an alarm and provide voice prompts to the user to take appropriate evasive action; the above are abnormal state judgments and decisions for several specific scenarios, and the embodiments of this invention include, but are not limited to, the above situations.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An FPGA-based intelligent guidance method for the blind, characterized in that, The steps are as follows: S1: The user calls the built-in blind navigation map and GPS / BeiDou navigation module in the helmet through the voice recognition module and voice prompt module, and sets the destination and departure point information; the blind navigation map plans the walking route according to the characteristics of the blind person and provides feedback to the user. After the user confirms, the navigation function is activated. S2: The binocular camera captures images from the front and back of the helmet in real time and transmits the images to the FPGA processing module; S3: The FPGA processing module performs multi-target detection algorithm on the input image to detect targets and performs binocular ranging algorithm to achieve binocular ranging. S4: The FPGA processing module performs hardware acceleration for the multi-target detection algorithm, and the acceleration work adopts a staged convolution operation acceleration method. S5: Uses an ultrasonic ranging module to accurately measure the distance to nearby obstacles; S6: Integrates navigation information, multi-target detection results, binocular ranging results, and ultrasonic ranging results, and provides detailed navigation voice prompts for the blind after analysis by a decision algorithm; S7: Adopt an adaptive energy-saving strategy and analyze the dynamic degree of the environment in all directions through the difference shadow method. When the image changes slightly, the multi-target detection and binocular ranging in step S3 will not be performed. The decision-making algorithm has the following steps: 1) Estimating target distance: Set the maximum ultrasonic ranging distance to M meters. When the ultrasonic ranging result is the maximum distance, the distances of each target measured by the binocular ranging algorithm are taken as the standard. When the ultrasonic ranging result is less than the maximum distance, it is determined that an obstacle exists. Then, the target closest to the ultrasonic ranging result is found in the binocular ranging result. This target is identified as the current obstacle, and its distance is D. If the target cannot be found in the binocular ranging result, the current obstacle is considered to be an unknown obstacle, and the ultrasonic ranging result is D. 2) Computational environment complexity: Where H represents the complexity of the environment, and N represents the number of targets. Indicates the rate of change of the target. This indicates the impact of target distance on the results; a1, a2, and a3 are the corresponding impact coefficients; 3) Determine whether to send a notification to the user based on the complexity of the environment: when the environmental complexity H is greater than or equal to the set value H t Send a reminder command if the message is not sent; otherwise, do not send a reminder command. 4) Determine the reminder type, and select N, and The one with the greatest impact is used as the alert type; when it is judged as N, the user is alerted that the surrounding environment is chaotic; when it is judged as When the environment is changing, it alerts the user; when it is determined to be... When the wearer is too close to surrounding targets, the device will alert the outside world to keep their distance from the user and activate the flashlight.

2. The FPGA-based intelligent guidance method for the blind according to claim 1, characterized in that, The multi-object detection algorithm will detect common objects in traffic scenes, and the binocular ranging algorithm will calculate the distance between the detected target objects and the camera. The specific implementation method is as follows: (1) The input image is processed by YOLO V5 neural network inference and nonmaximum suppression to generate target boxes; (2) The binocular ranging algorithm processes the input image to generate a depth map, and calculates the distance between each target and the camera based on the coordinates of the depth map and the target box; (3) The road image within the target box is semantically segmented using YOLO V5 to obtain the precise location of the road; (4) Analyze the information of the identified traffic lights and inform the current road traffic conditions.

3. The FPGA-based intelligent guidance method for the blind according to claim 2, characterized in that, The multi-target detection algorithm adopts the YOLO V5 algorithm and utilizes the FPGA processing module for hardware acceleration of convolution operations based on the Winograd algorithm. The tensor dimension transformation and nonlinear activation function operations of the output header of the YOLO V5 algorithm are processed on the CPU, while the convolution operations of other parts of the neural network are accelerated by the FPGA hardware. The convolution operations of other parts of the neural network are accelerated using an improved convolution operation hardware acceleration unit designed based on the Winograd algorithm.

4. The FPGA-based intelligent guidance method for the blind according to claim 3, characterized in that, The overall process of hardware acceleration of convolution operations based on the Winograd algorithm in the FPGA processing module is as follows: The FPGA processing module collects real-time information of the surrounding environment through two binocular cameras and stores the data in the memory; the relevant information of the neural network weights of the YOLO V5 algorithm for multi-object detection is stored in the memory in advance, and the Winograd transform convolution kernel is directly stored in the memory. The memory is configured with feature map cache and weight cache to store the input image and Winograd transform convolution kernel respectively; the programmable terminal inside the FPGA chip implements hardware acceleration of the convolution unit and pooling unit, and completes data transmission through DMA; after the hardware acceleration of the convolution layer and pooling layer of the neural network, the results are output to the CPU in the FPGA chip.

5. The FPGA-based intelligent guidance method for the blind according to claim 4, characterized in that, The method for hardware acceleration of convolution operations based on the Winograd algorithm is as follows: For an image with an input size of H×W, the image is divided into several image blocks, each with a size of h×w. The output parameter size is m×n, and the convolution operation has a kernel size of r×r. Y=A T [[GgG T ]⊙[B T dB]]A Where ⊙ represents the dot product of two matrices, i.e., multiplying corresponding elements; g represents a convolution kernel of size r×r; d represents an image patch of size (m+r-1)×(m+r-1); and G represents the kernel transformation matrix of size (m+r-1)×r, which is the Winograd transformation convolution kernel GgG. T B T A represents the input transformation matrix with size (m+r-1)×(m+r-1); T The output transformation matrix has a size of m×(m+r-1); where h=m+r-1, w=n+r-1; Y represents the result of the convolution operation; finally, the results obtained from all image blocks are stitched together.

6. The FPGA-based intelligent guidance method for the blind according to claim 4, characterized in that, The memory stores the Winograd transform convolution kernel. The convolution operation between the two-dimensional feature map d and the two-dimensional convolution kernel on the same channel in the memory is as follows: the two-dimensional feature map d and the Winograd transform convolution kernel stored in the memory on the same channel are respectively entered into two registers of a convolution operation acceleration unit via the AXI bus; then, the two-dimensional feature map d undergoes B transformation through the input feature map transformation module. T The dB operation yields the transformed feature map. Then, the transformed feature map and the Winograd transform convolution kernel are simultaneously fed into the matrix dot product operation unit to perform the ⊙ operation, outputting the result R. Finally, the result R is processed by the output feature map transformation module to obtain the result Y. The result Y is returned to the register and written back to the memory via the AXI bus. For accelerated convolution operations on a set of 3D feature maps and a 3D convolution kernel, a set of 3D feature maps and a 3D convolution kernel are retrieved from the feature map cache and weight cache, respectively. In the calculation of each channel of the set of 3D feature maps and the 3D convolution kernel, a convolution operation acceleration unit is used to complete the calculation in parallel, resulting in multiple 2D feature maps. An accumulation module adds the values ​​at the same positions of the multiple 2D feature maps to obtain a single feature map. Multiple sets of convolution operation acceleration units complete the calculation of a set of feature maps and a 4D convolution kernel in parallel, resulting in a 3D feature map as the output. The above process is performed on each set of input feature maps, and the final output results constitute a complete output feature map. The circuits of the input feature map transformation module and the output feature map transformation module both adopt the idea of ​​merging like terms in matrix operation results. For addition or subtraction operations of the same variables in a polynomial, a single adder or subtractor is used to implement matrix multiplication operations.

7. The FPGA-based intelligent guidance method for the blind according to claim 1, characterized in that, When the user is at a large intersection, the system detects the traffic and pedestrian flow and traffic light changes during that time period, and provides real-time prompts for the user to proceed according to traffic rules. If the user deviates from the normal pedestrian crossing or route, the system prompts the user to correct their course. When the user is on a tactile paving path with obstructions, the system detects the position and size of the obstructions and provides the information to the user via voice, allowing them to avoid the obstruction or change their course in advance. If an object approaches the user at high speed, a flashing light will sound an alarm, and a voice prompt will prompt the user to take appropriate evasive action.

8. The smart helmet system of the FPGA-based intelligent guidance method for the blind according to any one of claims 1-7, characterized in that, The helmet body includes an FPGA processing module (10), an ultrasonic ranging module (7), a binocular camera (8), a voice recognition module (2), a voice prompt module (1), a Bluetooth module (6), a flexible solar panel (4), a flashing warning module (9), a lithium battery and power management module (5), and a GPS / BeiDou navigation module (3). The FPGA processing module (10), ultrasonic ranging module (7), binocular camera (8), voice recognition module (2), voice prompt module (1), Bluetooth module (6), flexible solar panel (4), ultrasonic rangefinder module (7), binocular camera (8), voice recognition module (2), voice prompt module (1), Bluetooth module (6), flexible solar panel (7), and other components are equipped with an FPGA processing module (10), an ultrasonic ranging module (7), a binocular camera (8), a voice recognition module (2), a voice prompt module (1), a Bluetooth module (6), a flexible solar panel (4), a flashing warning module (9), a lithium battery and power management module (5), and a GPS / BeiDou navigation module (3). The flash warning module (9), lithium battery and power management module (5) and GPS / Beidou navigation module (3) are all installed on the helmet body. The flexible solar panel (4) is connected to the lithium battery and power management module (5). The ultrasonic ranging module (7), binocular camera (8), voice recognition module (2), voice prompt module (1), Bluetooth module (6), flash warning module (9), lithium battery and power management module (5) and GPS / Beidou navigation module (3) are all connected to the FPGA processing module (10). The Bluetooth module (6) is connected to the mobile terminal.

9. The smart helmet system according to claim 8, characterized in that, The flexible solar panel (4) covers the upper surface of the helmet body, and the GPS / Beidou navigation module (3) is placed directly above the helmet body for real-time positioning of the helmet body; The flash warning module (9) includes a flashlight, which is located on the rear side of the helmet body to alert pedestrians and oncoming vehicles. The helmet body is equipped with wide-angle binocular cameras at both the front and rear, and ultrasonic ranging modules (7) are installed on the front and sides of the helmet body. The voice recognition module (2) is connected to a microphone, which is located on the innermost side of the helmet body. The voice prompt module (1) broadcasts voice prompt information through a speaker. The FPGA processing module includes an FPGA chip with a built-in CPU core, a memory, a power management circuit, and an interface circuit. The FPGA chip is connected to the memory and the power management circuit respectively. The FPGA chip is connected to the ultrasonic ranging module (7), the binocular camera (8), the voice recognition module (2), the voice prompt module (1), the Bluetooth module (6), the flash warning module (9), the lithium battery and power management module (5), and the GPS / Beidou navigation module (3) respectively through the interface circuit. The lithium battery and power management module (5) includes at least a lithium battery and a multi-channel power management module. The lithium battery and power management module (5) is connected to the flexible solar panel (4). The FPGA chip is equipped with a multi-target detection unit and a binocular ranging unit to realize multi-target detection of images and target distance detection. The programmable end of the FPGA chip is equipped with a convolution operation acceleration unit for hardware acceleration. The sampling rate of the binocular camera is adaptively adjusted according to the environment. When the number of targets is stable and the target distance is safe, the FPGA processing module checks once every few frames.

Images