Zebra crossing accurate positioning method based on multiple sensors

Abstract

The invention discloses a multi-sensor-based zebra crossing accurate positioning method and device, and aims to solve the problems of low positioning accuracy and poor posture adaptability of existing blind navigation equipment. The equipment is in a chest hanging or handheld form, a six-axis IMU and a camera are arranged in the equipment, attitude data such as a pitch angle and a rolling angle and image information are collected, image inclination is corrected through a rotation compensation matrix, an aerial view is generated in combination with inverse perspective mapping, zebra crossings are identified by adopting a CNN and a target detection obb model, and a target detection target is obtained. And the actual distance between the blind person and the zebra crossing, the left-right offset and the length of the zebra crossing are converted. According to the invention, centimeter-level positioning is realized, equipment inclination and jitter errors are dynamically compensated, multi-dimensional guide information is output, algorithm robustness and environmental adaptability are improved, and accurate and safe travel assistance is provided for the blind to cross the street.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method for accurate zebra crossing positioning based on multi-sensor fusion. Background Technology

[0002] With the advancement of intelligent urban transportation and the construction of barrier-free travel, the safety of blind people traveling has received increasing attention. Currently, navigation for the blind mainly relies on voice navigation, electronic guide canes, or visual recognition assistive devices. These devices are generally based on GPS positioning and voice prompts, and can only provide macroscopic route navigation information. They are significantly insufficient in scenarios where positioning accuracy is extremely important, such as zebra crossings and intersections.

[0003] The shortcomings of existing technologies are mainly reflected in the following three aspects: GPS positioning accuracy is limited: Affected by factors such as obstruction, reflection, and multipath effect, GPS positioning error is usually on the order of several meters, making it difficult to provide the precise location information required at the pedestrian level. This results in blind people being unable to accurately perceive the location and direction of zebra crossings when crossing the street, posing a high safety risk. Pure visual recognition has poor robustness: Although existing camera-based visual recognition algorithms can detect zebra crossing areas, most of them rely on a fixed viewpoint and lack real-time correlation with human or device posture. They are prone to detection deviation or misjudgment due to factors such as changes in handheld angle, camera shake, and changes in lighting. Limitations of single sensor functionality: The maturity of inertial measurement unit (IMU) technology has made it possible to estimate the attitude of the device. A six-axis IMU can output pitch, roll and azimuth angles in real time, providing a stable reference coordinate system for the vision system. However, relying solely on the IMU cannot identify visual targets; while relying solely on the camera lacks a stable attitude reference and makes it difficult to obtain the precise position of the zebra crossing in space.

[0004] Therefore, how to effectively integrate a six-axis IMU with visual perception to achieve accurate identification and spatial positioning of zebra crossings in the visual field of blind people, and then calculate the actual accurate distance and left and right offset between the zebra crossing and the blind person, has become a key technical problem to improve the safety and autonomy of blind people crossing the street. This invention is a solution to this problem. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a multi-sensor-based method for precise zebra crossing positioning. By fusing data from a six-axis IMU and a camera, this method solves the problems of low positioning accuracy, poor attitude adaptability, and insufficient output information in existing technologies, achieving the following objectives: 1) Achieve centimeter-level relative positioning accuracy for zebra crossings; 2) Dynamically compensate for errors caused by device tilt and shaking, improving recognition stability under complex postures; 3) Output multi-dimensional effective information such as distance, offset direction, and length of zebra crossings to assist blind people in crossing the street accurately.

[0006] The core idea of ​​this invention is to acquire real-time attitude data of the device through a six-axis IMU, compensate for attitude errors in the images captured by the camera, convert the images into a bird's-eye view using inverse perspective mapping, and then identify zebra crossings using a target detection algorithm, ultimately achieving precise positioning in physical space. The specific steps are as follows: 1. Data Acquisition and Parameter Configuration (S1) The purpose of this step is to simultaneously acquire visual data and pose data, and configure personalized physical parameters to provide a foundation for subsequent calibration and localization. S1.1 Camera captures real-time environmental images (denoted as IMAGE1): The camera uses an image sensor with a resolution of no less than 1920×1080 and a frame rate of no less than 30fps to capture the ground scene in front of it in real time; S1.2 Six-axis IMU acquires attitude data: The IMU (such as MPU6050) includes a three-axis accelerometer and a three-axis gyroscope, with a sampling rate of not less than 100Hz, and outputs the device's attitude angles in real time, including: Pitch angle α: The angle between the device and the ground plane, which is the core factor affecting the accuracy of zebra crossing distance measurement; Roll angle θ: The angle of rotation of the equipment relative to the center, which affects the judgment of the zebra crossing direction; Yaw angle ψ: The left and right rotation angle of the equipment, which has little impact on positioning and is automatically corrected by the user's natural movement. S1.3 Configure device height from the ground: This setting is customized according to the user's height and wearing method. For example, if a user who is 1.7 meters tall wears the device on their chest, the height can be set to 1.5 meters; if the device is held in hand, the height can be set to 1.2-1.4 meters according to the user's grip habits.

[0007] 2. Image tilt correction based on roll angle (S2) To address the image tilt caused by the roll angle θ, a rotation-compensated affine matrix is ​​constructed for correction, ensuring the accuracy of zebra crossing direction determination. S2.1 Set the compensation rotation angle: In order to counteract the effect of the roll angle θ, set the compensation rotation angle θc=θ (opposite to the original attitude angle). S2.2 Determine the rotation reference point: The center point (cx, cy) of image IMAGE1 is used as the rotation reference, where... W is the image width (e.g., 1920 pixels), and H is the image height (e.g., 1080 pixels). S2.3 Construct the rotation compensation matrix: Construct the rotation matrix R(θc), expressed as: A translation matrix T and an inverse translation matrix T⁻¹ are introduced to eliminate the positional shift of the image during rotation. The expression for the translation matrix T is: The final rotation-compensated affine matrix M is generated, and its expression is: S2.4 Image Correction Execution: Call the cv2.warpAffine function in the OpenCV library, input the original image IMAGE1, the compensation matrix M, and the output image size (new_w=W, new_h=H), and obtain the tilt-corrected image IMAGE2.

[0008] 3. Inverse perspective mapping based on pitch angle (S3) To address the "perspective shortening" phenomenon of zebra crossings caused by the pitch angle α, inverse perspective mapping (IPM) is used to convert the corrected image IMAGE2 into a bird's-eye view, thus eliminating the impact of perspective distortion on distance measurement accuracy. S3.1 Determine the core parameters of perspective transformation: Camera intrinsic parameter matrix K: These are inherent parameters of the device, obtained from camera calibration, and are expressed as: Where f_x and f_y are the camera focal lengths, and c_x and c_y are the coordinates of the image center point; Rotation matrix R: contains camera attitude angles (pitch angle α, yaw angle ψ, roll angle θ), used to define the camera's viewpoint direction; Translation vector t: The position vector of the camera in the 3D world coordinate system, retaining only the ground height component ty (i.e., HEIGHT set in step S1.3), with all other components set to 0. The expression is: ; Ground normal vector n: When the ground is a horizontal plane (Y=0), n=[0,1,0]^T (representing the ground facing upwards); The distance d from the camera to the ground: that is, the height set in step S1.3; S3.2 Calculation of perspective transformation matrix: According to the standard perspective transformation homography formula H=K(Rt×n^T / d)K⁻¹, the inverse perspective mapping matrix H is calculated; S3.3 Bird's-eye view generation: Call the cv2.warpPerspective function in the OpenCV library, input the corrected image IMAGE2 and the perspective transformation matrix H, and output a bird's-eye view without perspective distortion. At this time, the zebra stripes in the image present the realistic features of equal width, equal spacing and parallelism.

[0009] 4. Zebra crossing detection in bird's-eye view (S4) By leveraging the features of zebra crossings in aerial views, accurate identification is achieved through a convolutional neural network (CNN) combined with an object detection OBB model. Feature extraction: Based on the characteristics of zebra crossings in the bird's-eye view, namely "equal width stripes, uniform spacing, and fixed direction", CNN models (such as YOLO series and ResNet) are used to extract image features; Object detection: The extracted features are processed using the standard object detection OBB (rotated bounding box) model to output the rotated bounding box of the zebra crossing area, and the pixel coordinates of the far center point P1 (the end away from the wearer) and the near center point P2 (the end closer to the wearer) of the zebra crossing are determined.

[0010] 5. Calculation of actual distance and orientation (S5) Converting pixel coordinates in a bird's-eye view into physical ground coordinates enables precise calculation of the relative position of zebra crossings: S5.1 Bird's-eye view physical resolution calibration: The "meter / pixel" ratio of the bird's-eye view is calculated by the homography matrix H. Based on the preset ground coverage area (e.g., 10 meters in front and 5 meters to the left and right) and the output image resolution, an orthophoto map with a fixed physical scale is constructed. S5.2 Pixel coordinate to physical coordinate mapping: Based on the calibrated physical resolution, the pixel coordinates (u1, v1) and (u2, v2) of the zebra crossing center points P1 and P2 are converted into ground physical coordinates (X1, Y1) and (X2, Y2) through linear mapping. S5.3 Key Parameter Calculation: The actual distance between a blind person and a zebra crossing: the straight-line distance from the projection point (X0, Y0) directly below the calculation device to P2 (X2, Y2), i.e., D=√[(X2-X0)²+(Y2-Y0)²]; Left and right offset: Calculate the horizontal deviation of P2(X2, Y2) from the axis directly in front of the equipment, i.e., ΔX=X2-X0 (positive value indicates rightward deviation, negative value indicates leftward deviation). Zebra crossing length: Calculate the straight-line distance between P1(X1,Y1) and P2(X2,Y2), i.e., L=√[(X1-X2)²+(Y1-Y2)²]; S5.4 Positioning Information Output: Outputs the zebra crossing's directional angle relative to the wearer's current orientation, actual distance, left and right offset, and zebra crossing length, such as "the zebra crossing is 2.3 meters ahead and 0.4 meters to the right, with a length of 5.0 meters," providing data support for voice guidance.

[0011] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages compared with the prior art: Significantly improved positioning accuracy: By adopting a multi-sensor fusion solution of a six-axis IMU and a camera, and through attitude compensation and geometric correction, the relative position of the zebra crossing is calculated with centimeter-level accuracy, which is significantly better than traditional GPS (with errors of several meters) or pure visual recognition solutions, effectively preventing blind people from accidentally entering the zebra crossing when the light is red. Strong environmental and posture adaptability: By using the posture data output in real time by the IMU, the device can be dynamically compensated for tilting and shaking during movement. It is not affected by changes in hand angle, wearing posture, or lighting, ensuring the stability of zebra crossing detection and improving the robustness and environmental adaptability of the algorithm. The output information is more comprehensive and practical: it can not only provide the distance and offset direction of the zebra crossing, but also output the accurate length of the zebra crossing, helping blind people estimate the time to cross the street, providing more effective information for accurately adjusting the direction of travel, and greatly improving travel safety and operability; Simple structure and easy to implement: For the error of yaw angle ψ, no additional hardware or complex algorithms are required. It can be automatically corrected by the user's natural movement, which simplifies the system design and reduces the implementation cost.

[0012] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall workflow of a multi-sensor-based method for precise zebra crossing positioning according to the present invention. Figure 2 This is a timing diagram of sensor data synchronous acquisition for a multi-sensor-based zebra crossing precise positioning method according to the present invention. Figure 3 This is a schematic diagram of the image tilt correction process for a multi-sensor-based zebra crossing precise positioning method according to the present invention. Figure 4 This is a timing diagram of the inverse perspective mapping process for a multi-sensor-based zebra crossing precise positioning method according to the present invention. Figure 5 The following is a flowchart illustrating the zebra crossing detection and positioning calculation process of a multi-sensor-based method for precise zebra crossing positioning according to the present invention; Figure 1; Figure 6 This is a schematic diagram of the attitude angle error processing strategy of the zebra crossing accurate positioning method based on multiple sensors according to the present invention; Figure 7 This diagram illustrates the relationship between the device configuration and parameter settings of a multi-sensor-based method for precise zebra crossing positioning according to the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be noted that the terms "vertical," "horizontal," "up," "down," "left," "right," and similar expressions used in this article are for illustrative purposes only and do not represent the only possible implementation.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0017] like Figure 1 As shown, after the device is started, it sequentially executes five core processes: data acquisition and parameter configuration (S1), image tilt correction (S2), inverse perspective mapping (S3), zebra crossing detection (S4), and distance and offset calculation (S5). In stage S1, the camera image (IMAGE1) is acquired, IMU attitude angles (pitch angle α, roll angle θ) are obtained, and the device's height above the ground is set, providing basic data for subsequent processing. S2 to S5 proceed step-by-step according to the logic of "error compensation - geometric transformation - target recognition - positioning calculation," ultimately outputting multi-dimensional positioning information including distance, left and right offset, and zebra crossing length. This process achieves seamless data integration through a closed-loop design, eliminating attitude errors at the source, overcoming the limitations of fragmented data processing in traditional solutions, ensuring centimeter-level positioning accuracy, and simultaneously outputting multi-dimensional information to meet the comprehensive needs of blind people crossing the street. S1 provides the data foundation, S2-S3 solve image distortion problems, S4 achieves accurate target recognition, and S5 completes physical parameter conversion. Each stage works together to ensure positioning stability and accuracy.

[0018] like Figure 2As shown, the system control module uniformly schedules the camera and six-axis IMU for synchronous data acquisition. The system control module first sends a start command to the camera to acquire the image IMAGE1, then sends an acquisition command to the six-axis IMU to obtain the pitch angle α and roll angle θ. Simultaneously, it reads the user-customized HEIGHT parameter settings and finally synchronizes and caches the image and attitude data through a timestamp alignment mechanism. During implementation, a camera frame rate of 30fps and an IMU sampling rate of 100Hz must be maintained to ensure that the spatiotemporal consistency error between attitude data and image data does not exceed 10 milliseconds. This timing design solves the problem of asynchronous data acquisition in existing technologies. By extracting core attitude parameters and adding personalized configurations, it reduces computational consumption while ensuring correction effectiveness. Image acquisition provides raw data for visual recognition, attitude data provides a benchmark for correction, synchronous caching ensures no data misalignment in subsequent processing, and the HEIGHT configuration adapts to different user needs, collectively laying the foundation for high-precision positioning.

[0019] like Figure 3 As shown, image tilt correction proceeds in the steps of "extracting parameters - setting compensation - constructing a matrix - performing correction". First, the roll angle θ is extracted from the IMU data, and a compensation rotation angle θc = θ is set to counteract the tilt effect. Then, using the image center point (cx = W / 2, cy = H / 2) as a reference, a rotation compensation matrix Rθc is constructed. A translation matrix T and an inverse translation matrix T⁻¹ are introduced to eliminate positional offset, generating the final rotation compensation affine matrix M. Finally, the cv2.warpAffine function is called to correct IMAGE1, outputting a tilt-free image IMAGE2. During implementation, it is necessary to ensure that the matrix construction follows trigonometric function rules, the translation matrix parameters are precisely matched with the image center point, and the output image size is consistent with the original image. This solution uses a combination of "reverse compensation + affine matrix" to accurately solve the problem of misjudgment of zebra crossing direction caused by equipment tilt. The correction process is based on pure mathematical calculations, which are fast and do not destroy the integrity of the zebra crossing area. This provides a high-quality image foundation for subsequent inverse perspective mapping. Each step sequentially realizes error source localization, compensation strategy determination, correction matrix construction and final image output, ensuring the correction effect in a progressive manner.

[0020] like Figure 4As shown, inverse perspective mapping eliminates perspective distortion through multi-parameter fusion. The IMU module inputs the pitch angle α to the image processing module, and the parameter configuration module provides HEIGHT and the camera intrinsic parameter matrix K. The image processing module reads the corrected IMAGE2, and calculates the mapping matrix H according to the standard perspective transformation homography formula by combining the rotation matrix R, the translation vector t=[0,HEIGHT,0]T, and the ground normal vector n=[0,1,0]T. Finally, the cv2.warpPerspective function is called to convert IMAGE2 into a distortion-free bird's-eye view. During implementation, the camera intrinsic parameter K needs to be determined in advance using the Zhang Zhengyou calibration method to ensure that the rotation matrix R is consistent with the actual attitude of the equipment and that the HEIGHT parameter is completely matched with the setting value in S1.3. This innovative solution combines attitude data, personalized parameters, and inherent device parameters to solve the problem that traditional perspective transformation ignores individual user differences and device attitude. The bird's-eye view makes the zebra crossings present realistic features of equal width and spacing, providing a consistent image basis for accurate ranging. Among them, the pitch angle α and HEIGHT are the core compensation parameters, and the matrix H is constructed to realize the correlation between pixels and physical space. The final output bird's-eye view clears the way for subsequent detection and calculation.

[0021] like Figure 5 As shown, the zebra crossing detection and localization calculation starts with an aerial view as input. First, a CNN extracts the core features of equal width, equal spacing, and parallelism of the zebra crossings. Then, an OBB (Object Detection Bypass) model identifies the zebra crossing region, determining the pixel coordinates of the far-end center point P1 and the near-end center point P2. Subsequently, the physical resolution (meters / pixel) of the aerial view is calculated using a homography matrix, linearly mapping the pixel coordinates to ground physical coordinates. The actual distance between the blind person and P2, the left-right offset ΔX, and the distance between P1 and P2 (zebra crossing length) are calculated sequentially. Finally, the information is integrated and converted into voice guidance output. During implementation, the CNN model needs to be optimized to improve feature recognition accuracy, ensuring that the physical resolution error does not exceed 0.01 meters / pixel, and that the linear mapping strictly adheres to the coordinate correspondence. This solution uses a combination of "CNN+obb model" to improve recognition accuracy in complex environments. It achieves accurate conversion between pixels and actual distance through physical resolution calibration. The three types of core parameters output comprehensively cover the needs of blind people crossing the street. Each step solves the problems of image feature extraction, target localization, spatial association, parameter calculation and information output, forming a complete positioning link and completely solving the pain point that blind people cannot perceive the spatial parameters of zebra crossings.

[0022] like Figure 6As shown, differentiated processing strategies are designed for three types of attitude angles: pitch angle α, roll angle θ, and yaw angle ψ. Pitch angle α, as a core factor affecting ranging accuracy, employs a combination of IMU data compensation and IPM mapping, using inverse perspective mapping to convert to a bird's-eye view and eliminate ranging errors. Roll angle θ affects the judgment of zebra crossing direction; an affine compensation matrix is ​​constructed to correct image tilt and restore the true direction. Yaw angle ψ only causes a slight shift, which is automatically corrected using "image prompts + natural user movement," requiring no additional hardware or algorithm compensation. During implementation, attitude angle thresholds need to be set: compensation intensity is increased when the absolute value of pitch angle α exceeds 10° and the absolute value of roll angle θ exceeds 5°; voice prompts are triggered when the zebra crossing deviates more than 10% from the center of the image. These three processing flows are executed in parallel. This differentiated strategy avoids the drawbacks of traditional "one-size-fits-all" compensation. Automatic correction of yaw angle ψ simplifies system design and reduces computing power consumption. The three processes work together to completely eliminate equipment attitude errors. Among them, pitch angle processing ensures ranging accuracy, roll angle processing ensures direction accuracy, and yaw angle processing improves ease of use, together providing an error control basis for high-precision positioning.

[0023] like Figure 7 As shown, the device supports both chest-mounted and handheld modes. Both modes incorporate a built-in six-axis IMU (three-axis accelerometer + three-axis gyroscope) and camera to ensure consistent core sensor performance. After selecting the mode based on user habits, users personalize the device's height above the ground according to their height. The reference range for the chest-mounted mode is 1.3-1.6 meters, and for the handheld mode, it's 1.0-1.4 meters. The device then initiates data synchronization, subsequent calibration, recognition, and positioning processes. During implementation, it's crucial to ensure stable sensor installation, a forward-facing camera, a fixed relative position between the IMU and camera, and consistent core parameters (camera resolution no less than 1920×1080, IMU sampling rate no less than 100Hz). This design overcomes the limitations of traditional single-mode devices. Built-in sensors ensure stable and synchronized data acquisition, personalized height configuration adapts to different users and scenarios, mode selection enhances practicality, standardized parameters guarantee consistent performance across different modes, and subsequent processes seamlessly integrate data acquisition and core processing, ensuring centimeter-level positioning and precise guidance in all modes.

[0024] Based on the core technologies of the multi-sensor-based zebra crossing precise positioning method and device disclosed above, and combined with the application scenarios and installation requirements of existing assistive devices for the blind, the following details the device's interaction with existing technologies, its linkage methods, installation details, and application logic for different usage scenarios: Scenario 1: Integration and linkage with traditional electronic guide canes Traditional electronic guide canes typically incorporate ultrasonic obstacle avoidance modules, enabling near-range obstacle detection, but lacking precise zebra crossing positioning capabilities. This device, through modular integration, complements the guide cane's functionality. Installation and connection: The device's camera and six-axis IMU module are embedded within a waterproof casing atop the head of a traditional electronic guide cane. The camera lens is exposed and tilted at a 30° angle to the cane head, facing directly forward. The six-axis IMU module and camera are kept relatively fixed (to prevent attitude data shift). Data exchange and power supply are achieved via a Type-C interface with the guide cane's main control board. The device's height above the ground is referenced in the attached diagram. Figure 7 The handheld shape parameters range (1.0-1.4 meters), which can be personalized by the user via a physical button on the side of the guide cane. Linkage details: The original ultrasonic obstacle avoidance module of the guide cane is responsible for obstacle detection within 1 meter; this device, according to the attached... Figure 2 The sensors synchronously acquire timing data, obtaining image and attitude data in real time, and then... Figure 3 Image tilt correction eliminates the effect of roll angle θ, and additional Figure 4 After eliminating perspective distortion caused by the pitch angle α through inverse perspective mapping, by attaching... Figure 5 The system calculates and outputs zebra crossing distance, left and right offset, and length data, which are then converted into voice prompts by the built-in voice broadcast module of the guide cane. When the ultrasonic module of the guide cane detects a nearby obstacle, the system first triggers an obstacle avoidance prompt (such as "There is an obstacle 0.5 meters ahead"). After the user avoids the obstacle, the system automatically switches to zebra crossing positioning and guidance mode, realizing the functional synergy of "near-distance obstacle avoidance + long-distance accurate street crossing", solving the pain point of traditional guide canes that "can avoid obstacles but are difficult to cross the street accurately".

[0025] Scenario 2: Integration with smart glasses for the blind Existing smart glasses for the blind mostly possess basic visual recognition and voice navigation functions, but lack a dynamic compensation mechanism for changes in device posture, making them prone to zebra crossing detection misalignment. Regarding installation and connection: the six-axis IMU module of this device is integrated into the inner side of the temple of the smart glasses (close to the temporal region to reduce wearing movement), and the camera is embedded in the center of the front of the glasses frame (the lens aligns with the user's line of sight). Both are connected to the main control chip of the glasses via a flexible cable, sharing power from the glasses' built-in lithium battery; the device's height from the ground is referenced in the attached diagram. Figure 7 The chest pendant's dimensions range from 1.3 to 1.6 meters, automatically configured by the user inputting their height via the accompanying mobile app. Linkage details: The smart glasses' original visual recognition module handles the detection of macroscopic targets such as pedestrians and road signs; this device, according to the attached... Figure 1The overall workflow involves simultaneously executing S1 data acquisition, S2-S3 image correction and mapping, and receiving ambient light sensor data from the smart glasses to dynamically adjust camera exposure parameters (such as reducing sensitivity in strong light and turning on the fill light in low light), thus compensating for the shortcomings of existing visual recognition algorithms affected by lighting conditions; (with attached...) Figure 6 After the attitude angle is differentiated, the zebra crossing positioning information is integrated with the GPS macro navigation instructions of the smart glasses, and the information is broadcast in a progressive manner according to the logic of "path guidance - zebra crossing positioning - street crossing time estimation". At the same time, the vibration module built into the temple of the glasses assists in feedback of the offset direction (vibrating the left temple when left-leaning, vibrating the right temple when right-leaning), forming a dual guidance of "voice + touch" to improve the accuracy of user perception.

[0026] Scenario 3: Integration with mobile navigation apps for the blind Existing navigation apps for the blind rely on GPS for route planning, but their positioning accuracy is insufficient in scenarios such as crossing streets and intersections, failing to provide specific location information for zebra crossings. Regarding installation and connection: This device is designed as an independent, portable attachment (compliant with the attached...). Figure 7 The device is designed as a chest pendant and connects to a smartphone via a USB-C interface, providing both power and data transmission. The camera is fixed facing forward, and a six-axis IMU module complements and verifies data with the phone's built-in gyroscope. Users input their height and the pendant's position (e.g., "center front" or "left side front") into a navigation app, and the app then... Figure 7 The parameter configuration logic automatically calculates and sends the HEIGHT parameter to this device. Linkage details: The navigation app provides macro-level route planning via GPS (e.g., "Walk to the intersection 50 meters ahead to cross the street"). When the user approaches the intersection (GPS positioning shows within 10 meters of the intersection), the app automatically triggers this device to start working, pressing the attached... Figure 2 Time-synchronized acquisition of images and pose data, followed by attachment Figure 3-4 After image processing, via attachment Figure 5 The device calculates the precise location of the zebra crossing, correcting GPS errors down to the meter level. The app matches the zebra crossing length data output by the device with the backend intersection width database to estimate the crossing time (e.g., "zebra crossing length 5.0 meters, estimated crossing time 7 seconds"). Simultaneously, it obtains the real-time status of the intersection's traffic lights (red / green / remaining time) via the mobile network. When the remaining green light time is greater than or equal to the crossing time and the device detects the zebra crossing, it announces, "Green light 12 seconds remaining, zebra crossing 2.5 meters ahead and 0.3 meters to the right, safe to cross." This achieves end-to-end guidance combining "macro-path navigation + precise zebra crossing positioning + traffic light coordination," solving the problem of existing navigation apps that "can point the way but struggle to accurately guide pedestrians."

[0027] Scenario 4: Integrated application with intersection traffic signal auxiliary systems Some city intersections have already deployed traffic signal assistive systems for the blind, which can announce light status via intersection broadcasts, but lack precise location guidance for individual users. Regarding installation and connectivity: This device adds a Bluetooth Mesh communication module (compatible with the communication protocol of existing traffic signal assistive systems) to establish a wireless connection with the signal transceiver in the intersection's traffic light control box. The camera and six-axis IMU module are attached... Figure 7 The device is worn around the user's neck as a pendant. After configuring the height parameter to the range of 1.3-1.6 meters, it synchronizes with the traffic light assist system via Bluetooth Mesh. The device is powered by a rechargeable lithium battery with a battery life of ≥8 hours and supports charging via USB-C. Linkage details: The traffic light assist system sends the current light status (red / green / yellow) and remaining time to this device in real time. This device operates according to the attached instructions. Figure 1 The process begins with data collection and processing, followed by... Figure 6 After processing the attitude angle error, if a zebra crossing is detected and the remaining green light time is greater than or equal to the estimated crossing time (based on the attached...) Figure 5 If the calculated zebra crossing length is obtained, the device will announce via its built-in voice module, "Green light has 10 seconds remaining. The zebra crossing is 3.0 meters ahead, with a length of 4.8 meters. It is safe to cross the street." If the light is red or the remaining time of the green light is insufficient, the device will guide the user to "Please wait at your current location. The zebra crossing is 3.0 meters ahead. The green light will turn on soon." Simultaneously, the device will feed back the user's real-time location data (derived from the zebra crossing location) to the traffic light assistance system. The system will adjust the intersection broadcast volume according to the user's location (the closer the user is to the zebra crossing, the louder the broadcast volume), achieving a linkage between "public intersection prompts + precise individual guidance," thereby improving the safety and convenience of blind people crossing the street.

[0028] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make various modifications and alterations without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the claims.

Claims

1. A method for precise zebra crossing location based on multiple sensors, characterized in that, Includes the following steps: S1: Read sensor information, specifically including: S1.1 Acquires real-time environmental images (IMAGE1) via a camera; S1.2 The device obtains its current attitude angles through a built-in six-axis IMU (including a three-axis accelerometer and a three-axis gyroscope), the attitude angles including pitch angle α, roll angle θ and yaw angle ψ; S1.3 Set the device height above the ground according to the user's height and the device's wearing position; S2: Construct a rotation compensation affine matrix based on the roll angle θ, perform tilt correction on image IMAGE1, and obtain calibrated image IMAGE2; S3: Based on the pitch angle α and the device height above the ground, geometric compensation is performed on the calibration image IMAGE2 through inverse perspective mapping (IPM) to map the ground area into a bird's-eye view; S4: Using a convolutional neural network (CNN) combined with an object detection OBB model, the zebra crossing area is identified in the bird's-eye view, and the pixel coordinates of the far-end center point P1 and the near-end center point P2 of the zebra crossing are determined. S5: Based on the camera's intrinsic and extrinsic parameters and the physical resolution of the bird's-eye view, linearly map the pixel coordinates of P1 and P2 to the ground physical coordinates, calculate the actual distance between the blind person and the starting point P2 of the zebra crossing, the left and right offset relative to the blind person's current orientation, and the actual length of the zebra crossing between P1 and P2.

2. The zebra crossing precise positioning method based on multiple sensors according to claim 1, characterized in that, The specific process of constructing the rotation compensation affine matrix in step S2 is as follows: S2.1 Set the compensation rotation angle θc=θ, where θc is used to counteract the image tilt caused by the roll angle θ; S2.2 The center point (cx, cy) of image IMAGE1 is used as the rotation reference point, where cx = W / 2, cy = H / 2, W is the image width, and H is the image height; S2.3 Construct the rotation compensation matrix R(θc), and introduce the translation matrix T and the inverse translation matrix T⁻¹ to generate the final rotation compensation affine matrix M; S2.4 calls the cv2.warpAffine function to perform a rotation transformation on IMAGE1 based on matrix M, and outputs the calibrated image IMAGE2.

3. The zebra crossing precise positioning method based on multiple sensors according to claim 1, characterized in that, The implementation of inverse perspective mapping in step S3 is based on the standard perspective transformation homography formula H=K(Rb)K, where: K is the camera intrinsic parameter matrix, which represents the inherent parameters of the device; R is a rotation matrix containing camera attitude angles (pitch angle α, yaw angle ψ, roll angle θ), used to determine the camera's viewing angle direction; t is the translation vector, the vector position of the camera in the 3D world coordinate system, retaining only the ground height ty value, with the other components being 0; The ground normal vector n = [0, 1, 0]ᵀ (when the ground is a horizontal plane Y = 0), and d is the distance from the camera to the ground plane, which is the HEIGHT set in step S1.3; The matrix mapping is completed by calling the cv2.warpPerspective function, which converts the calibrated image IMAGE2 into a bird's-eye view without perspective distortion.

4. The zebra crossing precise positioning method based on multiple sensors according to claim 1, characterized in that, In step S1, the pitch angle α is the core factor affecting the zebra crossing distance measurement accuracy, the roll angle θ is used to correct the zebra crossing direction judgment deviation, and the yaw angle ψ is automatically corrected by the user's natural movement without the need for additional attitude compensation operations.

5. The zebra crossing precise positioning method based on multiple sensors according to claim 1, characterized in that, In step S5, the physical resolution of the bird's-eye view is obtained by homography matrix calculation. The "meter / pixel" ratio is determined according to the preset ground coverage area and output image resolution to form an orthophoto map with a fixed physical scale.

6. The zebra crossing precise positioning method based on multiple sensors according to claim 1, characterized in that, The positioning information output in step S5 includes: the direction angle of the zebra crossing relative to the blind person's current orientation, the actual distance, the left and right offset, and the length of the zebra crossing. The left and right offset is output in the quantified form of "the zebra crossing is X meters ahead and slightly to the left / Y meters to the right".

7. The zebra crossing precise positioning method based on multiple sensors according to claim 1, characterized in that, The device can be worn as a chest pendant or held in the hand. The height of the device from the ground in step S1.3 is adjusted according to different wearing methods and the user's height.

Images