Blind writing aid teaching system, method and device

Abstract

The application relates to a blind writing auxiliary teaching system, method and device, which takes a virtual coordinate space as a unified medium, couples real-time three-dimensional position sensing of a writing tool with ordered key point decomposition of to-be-written content, establishes unified corresponding relationship between a real writing area and target content through coordinate mapping, generates a directional moving prompt according to a plane relative relationship between a current position of the writing tool and a next target key point, and outputs a pen lifting and falling prompt according to a deviation relationship between real-time height of the writing tool and a reference height, so that invisible writing tracks are converted into audible and followable step-by-step interactive guiding processes, and blind and low-vision users can independently complete writing training without visual comparison and continuous assistance of others.

Description

Technical Field

[0001] This invention belongs to the field of accessibility assistive technology, specifically relating to a writing-assisted teaching system, method, and device for the blind. Background Technology

[0002] Writing is one of the fundamental skills for human language expression and cultural transmission. For blind and visually impaired individuals, handwriting is not simply a matter of text input, but a complex and continuous learning process involving determining the starting position, perceiving the direction of movement, maintaining a continuous trajectory, memorizing the stroke order, and controlling the timing of putting down and lifting the pen. Traditional writing instruction relies heavily on the visual channel, with learners correcting handwriting deviations immediately by observing teacher demonstrations or comparing with model characters. However, blind and visually impaired individuals naturally lack this visual feedback pathway, making them prone to difficulties such as not knowing where to start writing, being unable to steadily follow the target trajectory, and having difficulty judging whether the pen tip has touched the writing surface. This severely restricts their ability to independently master paper writing skills. Developing a step-by-step interactive guidance scheme that can transform the invisible writing trajectory into an audible and followable one is of significant practical importance for ensuring educational equity and social participation for blind and visually impaired individuals.

[0003] Currently, existing accessibility technologies such as screen readers, voice input, and Braille keyboards primarily address the discrete access and input of digital content, failing to support the teaching needs of continuous writing skills on paper. While tactile embossed templates or hands-on assistance can provide limited shape and contour perception, they heavily rely on the presence of others or pre-made materials, lacking the ability to provide dynamic directional guidance and progressive feedback based on the user's real-time pen tip spatial position. Users struggle to complete independent, repeatable writing practice without external human support. Other solutions based on visual detection or depth cameras for 3D coordinate acquisition can locate the spatial position of writing tools, but these solutions focus on target detection accuracy or posture tracking itself, neglecting the entire writing learning process for blind users. This includes the orderly decomposition of the content to be written, the establishment of a unified correspondence between the writing area and the content, real-time planar directional guidance, and the determination of the timing of pen strokes in the vertical direction, thus failing to construct a complete interactive teaching process. Therefore, spatial positioning ability alone cannot naturally translate into a low-cognitive-burden, independently usable writing teaching method. Furthermore, in the depth measurement process of writing instruments, depth cameras often output transient abnormal values ​​that deviate significantly from the true depth on individual frames due to reasons such as hand shadows, directional reflection of infrared light on the surface of markers, or sensor thermal noise pulses. If these abnormalities are not effectively suppressed, they will directly affect the accurate determination of the pen tip contact state downstream, causing misleading interference for blind users who rely on auditory feedback.

[0004] Therefore, an optimized teaching scheme for Braille writing assistance is desired. Summary of the Invention

[0005] This invention application provides a Braille writing assistance teaching system, method, and apparatus. According to one aspect of this application, a Braille writing assistance teaching method is provided, comprising: Step 1: Detect writing pen markers and obtain 3D coordinates from the collected writing area data to get the marker center coordinates and marker depth values; Step 2: Based on the boundary points of the writing area, normalize the coordinate mapping of the marker center and the marker depth value and perform contact height calibration to obtain virtual space mapping parameters, marker calibration height and height fluctuation threshold; Step 3: Extract and resample key points of the content to be written in an orderly manner, and map them to the virtual coordinate space using virtual space mapping parameters to obtain the target key point sequence; Step 4: Use virtual space mapping parameters to map the center coordinates of the marker frame by frame to obtain the mapping result. Subtract the mapping result from the current key point in the target key point sequence and calculate the orientation angle and distance. At the same time, quantize the height difference based on the marker depth value and the marker calibration height to obtain the planar orientation angle, arrival distance and real-time height difference. Step 5: Based on the division of the direction interval, the plane direction angle is quantified and throttling stability is determined to generate plane movement prompts. The real-time height difference is judged according to the arrival distance and height fluctuation threshold to generate pen lifting and lowering prompts. When the arrival distance meets the arrival threshold, the current key point is switched to the subsequent key point in the target key point sequence until all key points are reached.

[0006] According to another aspect of this application, a Braille writing-assisted teaching system is provided, comprising: The marker detection and depth acquisition module is used to detect writing pen markers and acquire three-dimensional coordinates from the collected writing area data to obtain the marker center coordinates and marker depth values; The coordinate mapping and height calibration module is used to perform normalized coordinate mapping and contact height calibration on the marker center coordinates and marker depth values ​​based on the writing area boundary points to obtain virtual space mapping parameters, marker calibration height, and height fluctuation threshold. The key point extraction and spatial mapping module is used to extract and resample key points of the content to be written in an orderly manner, and to map them to a virtual coordinate space using virtual spatial mapping parameters to obtain the target key point sequence. The real-time writing deviation calculation module is used to map the center coordinates of the marker frame by frame using virtual space mapping parameters to obtain the mapping result. The mapping result is subtracted from the current key point in the target key point sequence and the orientation angle and distance are calculated. At the same time, the height difference is quantized according to the marker depth value and the marker calibration height to obtain the planar orientation angle, the distance to reach and the real-time height difference. The interactive prompt generation and iteration control module is used to quantify and throttle stability of the plane direction angle based on the direction interval division to generate plane movement prompts, determine the pen lifting and lowering status of the real-time height difference based on the arrival distance and height fluctuation threshold to generate pen lifting and lowering prompts, and switch the current key point to the subsequent key point in the target key point sequence when the arrival distance meets the arrival threshold until all key points have been reached.

[0007] Compared with existing technologies, this invention proposes a writing assistance teaching system, method, and device for the blind. Using a virtual coordinate space as a unified medium, it couples the real-time three-dimensional position perception of the writing tool with the ordered key point decomposition of the content to be written. Through coordinate mapping, a unified correspondence is established between the real writing area and the target content. Then, based on the planar relative relationship between the current position of the writing tool and the next target key point, directional movement prompts are generated. Simultaneously, based on the deviation between the real-time height of the writing tool and the reference height, pen-starting and-stopping prompts are output. This transforms the invisible writing trajectory into an audible and followable step-by-step interactive guidance process, enabling blind and low-vision users to independently complete writing training without visual comparison or continuous assistance from others. This solution solves the problem of users not knowing where to start writing and being unable to steadily advance along the target trajectory by breaking down the content to be written into a sequence of key points with stroke order and guiding them point by point. It also solves the problem of users having difficulty perceiving whether the pen tip is in contact with the writing surface through a vertical height difference determination mechanism. Furthermore, by introducing inter-frame temporal constraints and adaptive fusion strategies in the depth calculation stage, it suppresses the interference of transient noise from the depth sensor on pen-starting and-stopping determination, ensuring the temporal consistency and physical rationality of the auditory guidance signal. Attached Figure Description

[0008] Figure 1 A flowchart of a blind writing-assisted teaching method according to an embodiment of this application; Figure 2 This is a schematic diagram of data flow in a blind writing-assisted teaching method according to an embodiment of this application; Figure 3 This is a schematic diagram of a camera acquisition and marker detection scenario in a blind writing-assisted teaching method according to an embodiment of this application; Figure 4 This is a flowchart of step one of the blind writing-assisted teaching method according to an embodiment of this application; Figure 5This is a flowchart of step 1.4 of the blind writing-assisted teaching method according to an embodiment of this application; Figure 6 This is a flowchart of step four of the blind writing-assisted teaching method according to an embodiment of this application; Figure 7 This is a schematic diagram illustrating the directional sound path and stroke guidance of the blind writing assistance teaching method according to an embodiment of this application; Figure 8 This is a schematic diagram of the system operation interface and writing results of the blind writing-assisted teaching method according to an embodiment of this application; Figure 9 This is a block diagram of a blind writing-assisted teaching system according to an embodiment of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0010] To address the challenges faced by blind and visually impaired users who lack visual feedback channels and are unable to independently learn paper writing, the lack of real-time dynamic guidance in existing tactile templates or manual assistance methods, and the failure of existing 3D positioning technology to form a complete interactive loop for writing instruction, this solution addresses these issues through the following implementation process: First, a depth camera is used to acquire color and depth images of the writing area. A target detection model identifies detectable markers on the writing pen and calculates their center coordinates and depth values ​​to obtain the real-time 3D spatial position of the writing tool. Second, during the calibration phase, the coordinates of the boundary points of the writing area and the reference height when the pen tip contacts the writing surface are recorded. Based on this, a system is established from the physically acquired coordinates to the virtual coordinates. The normalized mapping relationship of the coordinate space and the height floating threshold enable the writing behavior under different paper sizes and camera arrangements to be uniformly represented. Subsequently, the content to be written is transformed into a sequence of target key points with a writing order through skeleton refinement and stroke order analysis, and mapped to the same virtual coordinate space. In the formal writing stage, the system maps the coordinates of the markers to the virtual space frame by frame, calculates the direction angle and distance between the current position and the next target key point, and generates planar movement voice prompts after direction interval quantization and throttling stabilization determination. At the same time, the real-time depth of the markers is subtracted from the calibrated height and compared with the floating threshold to generate pen start and stop prompts, progressing step by step until all strokes are completed. In addition, to address the problem of abrupt changes in depth values ​​in individual frames due to occlusion or reflection of the depth camera, this solution introduces an inter-frame temporal smoothing prediction and adaptive fusion mechanism in the depth calculation stage. The fusion weight is dynamically adjusted according to the degree of deviation of the measured depth of the current frame from the historical trajectory, effectively suppressing the transmission of transient noise pulses to the downstream pen start and stop determination, and ensuring the temporal continuity and physical rationality of the auditory guidance signals received by blind users.

[0011] Figure 1 This is a flowchart of a blind writing-assisted teaching method according to an embodiment of this application. Figure 2 This is a schematic diagram of the data flow in a blind writing-assisted teaching method according to an embodiment of this application. Figure 1 and Figure 2As shown, the blind writing-assisted teaching method according to an embodiment of this application includes the following steps: Step 1, performing writing pen marker detection and three-dimensional coordinate acquisition on the collected writing area data to obtain the marker center coordinates and marker depth values; Step 2, based on the writing area boundary points, performing normalized coordinate mapping and contact height calibration on the marker center coordinates and marker depth values ​​to obtain virtual space mapping parameters, marker calibration height, and height fluctuation threshold; Step 3, performing ordered extraction and resampling of key points in the content to be written, and mapping it to a virtual coordinate space using the virtual space mapping parameters to obtain a target key point sequence; Step 4, using the virtual space mapping parameters to... The center coordinates of the marker are mapped frame by frame to obtain the mapping result. The difference between the mapping result and the current key point in the target key point sequence is calculated, and the orientation angle and distance are solved. At the same time, the height difference is quantized according to the marker depth value and the marker calibration height to obtain the plane orientation angle, arrival distance and real-time height difference. Step 5: The plane orientation angle is quantized and throttling stability is determined based on the orientation interval division to generate a plane movement prompt. The pen lifting and lowering status is determined according to the arrival distance and height fluctuation threshold to generate a pen lifting and lowering prompt. When the arrival distance meets the arrival threshold, the current key point is switched to the subsequent key point in the target key point sequence until all key points have been reached.

[0012] Specifically, in step one, the collected writing area data is processed to detect writing pen markers and obtain 3D coordinates to get the marker center coordinates and marker depth values. It is worth noting that the collected writing area data includes color image frames of detectable markers on the writing pen and alignment depth image frames.

[0013] Figure 3 This is a schematic diagram of a camera acquisition and marker detection scenario in a blind writing assistance teaching method according to an embodiment of this application. Figure 3As shown, the image and depth acquisition module uses an Intel RealSense D435 depth camera, which is mounted directly above the writing area with the lens optical axis pointing vertically towards the writing surface. The camera's installation height is set within the range of 40 to 60 centimeters from the writing surface to ensure that the entire effective area of ​​the writing area (A4 paper size) falls completely within the camera's field of view coverage. The camera's color sensor and depth sensor work synchronously at a fixed frame rate of 30 frames per second. The resolution of the color image frame is set to 640×480 pixels, and the resolution of the depth image frame is also set to 640×480 pixels. The camera synchronously outputs a color image frame containing markers and an aligned depth image frame. Furthermore, the depth image frame and the color image frame are spatially aligned pixel by pixel using the hardware alignment function built into the RealSense SDK. This allows any pixel coordinate position in the color image frame to directly index the depth measurement value at the same position in the aligned depth image frame without the need for additional coordinate system transformation calculations. The detectable marker is a 3D-printed, high-saturation purple ring fitted onto the pen barrel. This ring has an outer diameter of 12 mm and a width of 8 mm, and features a matte surface to reduce interference from high-gloss reflections on detection stability. The ring is positioned approximately 30 mm from the tip of the pen to ensure it is not completely obscured by the user's fingers when holding the pen normally and remains within the camera's field of view. The visual target detection model identifies the marker region in the color image frame and outputs the marker's bounding box. Subsequently, based on the bounding box, the geometric center coordinates of the marker are calculated, and a local depth sampling window is cropped at the corresponding position in the aligned depth image frame for subsequent depth value calculation.

[0014] It is understandable that, since the writing assistance teaching system for the blind needs to continuously acquire the real-time spatial position information of the writing pen during the user's writing process, and the writing pen itself is a slender rod-shaped object, it lacks stable and unique visual anchoring features in the image. Directly locating the pen body will face problems such as occlusion sensitivity, edge blurring, and posture ambiguity. At the same time, the subsequent virtual coordinate mapping and orientation angle calculation of the system rely on accurate two-dimensional center coordinates and reliable spatial depth values ​​as the calculation basis. Therefore, in the technical solution of this application, the writing pen marker detection and three-dimensional coordinate acquisition are performed on the collected writing area data to obtain the marker center coordinates and marker depth values. This provides a stable and accurate spatial position data source for downstream normalized coordinate mapping, contact height calibration, frame-by-frame orientation angle calculation, and pen lifting and lowering state determination. In this way, the continuous motion trajectory of the writing pen in physical space can be transformed into a structured numerical sequence with sub-pixel-level planar positioning accuracy and millimeter-level depth resolution, ensuring that the system's perception of the pen tip position has frame-level real-time performance and spatial robustness, thereby laying a data foundation for the accuracy of the directional guidance and pen lifting and lowering prompts received by blind users.

[0015] Figure 4 This is a flowchart of step one of the blind writing-assisted teaching method according to an embodiment of this application. Figure 4 As shown, step one includes: Step 1.1, inputting the color image frame into the pre-trained visual object detection model to perform bounding box regression and region recognition on the detectable markers on the writing pen to obtain the marker bounding box; Step 1.2, based on the coordinates of the diagonal vertices of the marker bounding box, calculating the geometric center of the bounding box region to obtain the marker center coordinates; Step 1.3, taking the marker center coordinates as the center, cropping a local window of a preset pixel size on the aligned depth image frame, and removing zero-value invalid values ​​and statistical outliers from the depth measurement values ​​of each pixel within the window to obtain an effective depth set; Step 1.4, based on each pixel and the marker center coordinates, performing depth-weighted fusion on the effective depth set to obtain the marker depth value.

[0016] Accordingly, in step 1.1, the color image frame is input into a pre-trained visual object detection model to perform bounding box regression and region recognition on the detectable markers on the writing pen to obtain the marker bounding boxes. It should be understood that since the color image frames in the writing area acquisition data contain mixed information from various visual elements such as the writing pen, the user's hand, the paper background, and the desktop environment, the system needs to quickly and accurately locate the pixel region occupied by the detectable markers on the writing pen in this complex scene. The appearance of the markers varies in scale and shape under different pen-holding postures, lighting conditions, and degrees of occlusion. Simple color thresholding or template matching methods are difficult to maintain stable localization performance under these variable conditions. Therefore, in the technical solution of this application, the color image frame is input into a pre-trained visual object detection model to perform bounding box regression and region recognition on the detectable markers on the writing pen to obtain the marker bounding boxes. This allows for the accurate delineation of the marker region within a single frame image using end-to-end inference, providing a pixel coordinate range with clear spatial boundaries for subsequent geometric center calculation. This enables the system to maintain stable detection of markers under actual writing conditions such as user hand occlusion, pen tilt, and changes in lighting. It transforms the problem of marker image localization from uncontrollable manual feature engineering into a data-driven robust regression problem, ensuring that the marker bounding box output in each frame maintains a high degree of spatial consistency with the imaging area of ​​the physical entity of the marker.

[0017] More specifically, in a concrete example of this application, the visual object detection model adopts the YOLOv5s network architecture, which consists of a CSPDarknet53 backbone network, a PANet feature pyramid neck network, and a three-scale detection head. First, the current frame of the color image is scaled proportionally from its original resolution of 640×480 pixels and padded to the model's standard input size of 640×640 pixels. During the scaling process, a bilinear interpolation algorithm is used to maintain the spatial continuity of the marker texture features. The padded areas are filled with a grayscale value of 114 to avoid introducing artificial boundary features that bias the detection results. Next, the preprocessed image data is normalized by channel dimension, linearly mapping the pixel values ​​from the integer range of 0 to 255 to the floating-point range of 0 to 1 before being fed into the CSPDarknet53 backbone network. This backbone network performs multi-level feature extraction on the input image through stacked convolutional layers, batch normalization layers, and SiLU activation function layers. It outputs three sets of multi-scale feature maps with strides of 8, 16, and 32 at layers 8, 16, and 24, respectively. The feature map with a stride of 8 preserves the fine-grained texture and edge information of the markers, while the feature map with a stride of 32 captures the global semantic contrast relationship between the markers and the background. Then, the three sets of multi-scale feature maps are fused bidirectionally from top to bottom and bottom to top through the PANet neck network. Through lateral connections and feature concatenation operations, deep semantic information and shallow spatial detail information are aggregated across scales to generate three sets of fused and enhanced multi-scale detection feature maps. Furthermore, the three-scale detection head performs bounding box offset regression and target class confidence prediction at each spatial location on the three sets of detection feature maps, using the preset anchor box size as a reference basis. It outputs the center coordinate offset, width and height scaling factors, and corresponding marker class confidence scores of all candidate detection boxes. Then, non-maximum suppression is performed on all candidate detection boxes to eliminate overlapping redundant boxes. Finally, the single detection result with the highest confidence score is retained as the marker bounding box output. The marker bounding box is represented by a quadruple of the pixel coordinates of the upper left corner vertex and the lower right corner vertex, which represents the rectangular area occupied by the marker in the current color image frame.

[0018] Specifically, the training process of this visual target detection model is as follows: First, images of the markers are collected under different lighting intensities (natural light, side lighting from a desk lamp, and top lighting from a fluorescent lamp), different shooting distances (ranging from 35 cm to 65 cm), different writing backgrounds (white paper, grid paper, and wood-grain desktop), different paper materials (ordinary A4 paper, cardstock, and grid calligraphy paper), different writing postures (various angles from vertical to 45-degree forward tilt of the pen), different degrees of occlusion (10% to 40% of the marked ring area is covered by fingers), and different user pen grip methods (three-finger grip, four-finger grip, and left-hand grip). A total of 3,000 original images are collected. Bounding boxes, center points, masks, and key regions are annotated on all the original marker images, with the annotation boxes tightly surrounding the visible outer contour of the marked ring. Data augmentation was performed on the labeled images. Augmentation strategies included random brightness perturbation (±30%), random rotation (±15 degrees), random scaling (0.8 to 1.2 times), random cropping (preserving 70% to 100% of the original image area), Gaussian blur (kernel size randomly selected from 3×3 to 7×7), random rectangular region occlusion (occlusion area not exceeding 20% ​​of the labeled area), and random background replacement (randomly sampling from a pre-set library of 50 different desktop textures and synthesizing them after perspective transformation). After data augmentation, the training set was expanded to 15,000 valid samples. The augmented training data was then fed into the YOLOv5s object detection model for training. The input image size was uniformly scaled to 640×640 pixels, the training batch size was set to 16, the initial learning rate was set to 0.01, and a cosine annealing scheduling strategy was adopted. The total number of training epochs was 300, and the loss function was a weighted combination of CIoU bounding box regression loss and binary cross-entropy classification loss. After training, the model achieved an average accuracy (mAP@0.5) of 98.6% for marker detection on the validation set. The single-frame inference time on the NVIDIA Jetson Nano embedded platform was less than 25 milliseconds, meeting the real-time detection frame rate requirement of 30 frames per second, thus obtaining a detection model for real-time identification of detectable markers.

[0019] Accordingly, in step 1.2, the geometric center of the bounding box region is calculated based on the coordinates of its diagonal vertices to obtain the marker center coordinates. It should be understood that since the marker bounding box output in the previous sub-step is a rectangular region described by the coordinates of its top-left and bottom-right diagonal vertices, it only defines the area occupied by the marker in the image, not a unique spatial anchor point. Subsequent depth sampling window truncation, virtual coordinate mapping, and planar orientation angle calculation all require a single two-dimensional pixel as the computational basis, and the marker bounding box cannot directly participate in the aforementioned point-by-point calculations. Therefore, in the technical solution of this application, the geometric center of the bounding box region is further calculated based on the coordinates of its diagonal vertices to obtain the marker center coordinates. This compresses the rectangular region description into a uniquely determined two-dimensional pixel coordinate point, providing a precise spatial reference origin for subsequent depth sampling window positioning and affine transformation mapping. In this way, the system can extract the pixel position representation that is consistent with the centroid height of the projected physical entity of the marker from the geometric properties of the bounding box with minimal computational overhead, ensuring the spatial continuity of the coordinate point and the smoothness of the motion trajectory between different frames, while avoiding systematic offsets introduced by using a vertex or edge point of the bounding box as a reference.

[0020] More specifically, in a concrete example of this application, the bounding box of the marker is represented by a quadruple coordinate system output by the visual object detection model. The coordinates of the top-left vertex record the minimum pixel index values ​​of the marker-occupied area in the image, both horizontally and vertically, while the coordinates of the bottom-right vertex record the maximum pixel index values ​​in both directions. First, the x-coordinates of the top-left and bottom-right vertices of the bounding box are extracted, and their arithmetic mean is calculated to obtain the midpoint position of the bounding box in the horizontal dimension. Next, the y-coordinates of the top-left and bottom-right vertices are extracted, and their arithmetic mean is calculated to obtain the midpoint position of the bounding box in the vertical dimension. Then, the horizontal and vertical midpoint positions are combined into a two-dimensional coordinate tuple, which constitutes the complete pixel coordinate representation of the marker's center coordinates. In the actual operation scenario of Braille writing-assisted teaching, since the detectable marker is a 3D printed high-saturation purple marker ring fitted onto the pen barrel, the marker ring has an axisymmetric circular geometric structure. Its orthographic projection on the image plane is approximately elliptical. The geometric center of this ellipse is highly consistent with the projection point of the axis of the physical entity of the marker ring on the imaging plane. Therefore, the coordinates of the marker center obtained by the arithmetic mean of the diagonal vertices of the bounding box can accurately reflect the projection position of the physical centroid of the marker ring on the image plane. This positional deviation does not exceed 1 pixel within the normal pen-holding posture range, which meets the requirements of subsequent virtual coordinate mapping for planar positioning accuracy.

[0021] Accordingly, in step 1.3, a local window of a preset pixel size is cropped from the aligned depth image frame, centered on the marker center coordinates. Zero-value invalidities and statistical outliers are removed from the depth measurements of each pixel within the window to obtain a valid depth set. It should be understood that the depth value of a single pixel corresponding to the marker center coordinates in the aligned depth image frame is easily affected by factors such as structured light projection noise from the depth camera, changes in the microscopic reflectivity of the marker surface, and sensor quantization errors, resulting in instantaneous deviations. Relying solely on the depth reading of a single pixel cannot provide a sufficiently stable numerical basis for subsequent height difference calculations. Furthermore, the aligned depth image frame contains zero-value invalid measurement points caused by hand obstruction of the infrared structured light or specular reflection from the marker surface, as well as statistical outliers caused by multipath interference or edge depth jumps. If these abnormal data are not removed, they will directly contaminate the depth fusion result. Therefore, in the technical solution of this application, a local window of a preset pixel size is cropped from the aligned depth image frame, centered on the coordinates of the marker center. Zero-value invalid values ​​and statistical outliers are removed from the depth measurement values ​​of each pixel within the window to obtain an effective depth set. This extracts a set of reliable depth samples that have undergone quality screening from the neighborhood of the marker center, providing noise-free input data for subsequent weighted fusion calculations. In this way, single-pixel depth sampling can be extended to multi-pixel region sampling, and a dual removal mechanism can filter out physically invalid values ​​and statistical outliers. This ensures that each element in the remaining effective depth set is within a reasonable depth measurement range and statistically consistent with each other, thereby eliminating noise interference at the data source level for stable calculation of marker depth values.

[0022] More specifically, in a particular example of this application, let the center position of the marker be... ,in The x-coordinate of the marker center indicates the time of data collection. The vertical axis is The horizontal pixel size of the depth sampling region is Vertical pixel size is The mathematical definition of the depth sampling region is: in, This represents the rectangular depth sampling area constructed at acquisition time t with the center coordinates of the marker as the anchor point; This represents the x and y coordinates of any pixel in the aligned depth image frame that falls within this sampling region. This represents the horizontal pixel coordinates of the marker's center at acquisition time t; The vertical pixel coordinates of the marker center at acquisition time t; Indicates the horizontal pixel size of the depth sampling region; This represents the vertical pixel size of the depth sampling region. In this example, and Each value is fixed at 5, forming a 5×5 pixel region. This formula clearly defines the spatial geometric range of the sampling region through absolute value inequality constraints, where... The horizontal sampling radius is constrained to no more than 2 pixels. The vertical sampling radius was constrained to no more than 2 pixels. Together, these constraints determined a 25-pixel rectangular neighborhood centered on the marker's center coordinates. The size of this neighborhood balanced the statistical sufficiency of the sampling coverage area with the spatial compactness of avoiding background depth contamination. In this example, with a marker ring outer diameter of 12 mm and the camera 50 cm from the writing surface, the projection diameter of the marker ring in the image is approximately 15 to 18 pixels. The 5×5 sampling window falls entirely within the marker's projection area without capturing the background depth outside the marker ring.

[0023] First, the depth sampling region is defined according to the above formula. The depth measurements of these 25 pixel locations are read from the aligned depth image frame to form the original depth sampling set. Next, a zero-value invalidation operation is performed on the original depth sampling set. All 25 depth measurements in the set are traversed, and elements with values ​​strictly equal to zero are marked as invalid and removed from the set. This is because the depth camera outputs zero values ​​as invalid markers at pixel locations where no valid infrared structured light echo signal is received. This situation occurs when the infrared echo energy is insufficient in the edge region of the marker ring due to the surface normal direction deviating from the camera's optical axis, and when the infrared projection area is blocked by the user's finger. After zero-value invalidation, a non-zero depth candidate set is obtained. Then, a statistical outlier removal operation based on interquartile range is performed on the non-zero depth candidate set. All elements in the non-zero depth candidate set are sorted in ascending order of numerical value. The value at the 25th percentile of the sorted sequence is calculated as the first quartile, and the value at the 75th percentile is calculated as the third quartile. The difference between the two is the interquartile range. Subsequently, elements whose depth measurement value is less than the first quartile minus 1.5 times the interquartile range or greater than the third quartile plus 1.5 times the interquartile range are identified as statistical outliers and removed from the set. This criterion is based on the box plot anomaly detection principle and can effectively identify outlier measurements caused by multipath interference, depth jumps between marker edges and background, or transient electronic noise pulses from sensors without relying on prior assumptions about depth distribution. Furthermore, after two rounds of screening—zero-value invalidation and statistical outlier removal—all the remaining depth measurements constitute the effective depth set. Each element in this set corresponds to a spatial distance measurement sample with physical validity and statistical consistency within the depth sampling area. Under typical working conditions in this example, the number of elements retained in the effective depth set after double removal is between 18 and 24. This sample size is sufficient to support the statistical stability requirements of subsequent fusion operations.

[0024] Accordingly, in step 1.4, depth-weighted fusion is performed on the effective depth set based on the coordinates of each pixel and the marker center to obtain the marker depth value. It should be understood that although the multiple depth elements retained in the effective depth set have all passed quality screening by zero-value removal and outlier removal, these elements are spatially distributed at different pixel positions in the neighborhood of the marker center. Pixels farther from the marker center coordinates are more susceptible to the marker torus edge effect, the attenuation of the spatial resolution of the depth sensor, and depth crosstalk from adjacent background regions. If a simple arithmetic mean operation is performed on all effective depth elements, edge pixels and center pixels are given the same contribution weight, failing to reflect the higher reliability of the spatial prior characteristics of the depth measurement value in the central region. Therefore, in the technical solution of this application, depth-weighted fusion is further performed on the effective depth set based on the coordinates of each pixel and the marker center to obtain the marker depth value. This allows for differentiated allocation of reliability weights according to the spatial distance relationship between the pixel position of each effective depth element and the marker center coordinates, resulting in a larger contribution of the high-reliability measurement value in the central region to the fusion result. In this way, while retaining the advantages of multi-pixel statistical smoothing, spatial confidence gradient is introduced, making the final output marker depth value closer to the true spatial depth of the physical center area of ​​the marker, providing a more accurate vertical positioning benchmark for subsequent height difference calculation and pen lifting and lowering state determination.

[0025] More specifically, in a concrete example of this application, firstly, for each valid depth element in the valid depth set, the planar Euclidean distance between the pixel coordinates corresponding to that element and the coordinates of the marker center is calculated. Based on the reciprocal of this distance, a confidence weight is assigned, with pixels closer to the marker center coordinates being assigned a higher confidence weight value, and pixels farther away being assigned a lower weight value. Next, a normalized weighted summation operation is performed on all valid depth elements in the valid depth set and their corresponding confidence weights. The calculation of the marker depth value is expressed as: in, This represents the marker depth value calculated at acquisition time t; This represents the j-th valid depth value in the valid depth set; Represents the weights related to pixel location, detection confidence, and depth stability; It represents the lower bound of the index of the valid depth element in the valid depth set; This represents the upper bound of the index of the valid depth element in the valid depth set. In this example, the weight... The assignment is based solely on the planar Euclidean distance between each effective depth element and the marker center coordinates; that is, pixels closer to the center are assigned higher confidence weights. The numerator in the above formula... This represents the summation of element-wise products of all valid depth values ​​within the valid depth set and their corresponding weight coefficients. Its physical meaning is a weighted accumulation of the depth distribution in the neighborhood of the marker, using spatial confidence as a modulation factor; the denominator is... This represents the normalized base of all weight coefficients, ensuring that the dimensions of the fusion result are consistent with those of a single depth measurement. Then, the result of the above normalized weighted summation operation is directly output as the marker depth value for the current frame.

[0026] Specifically, it should be noted that in the marker depth calculation stage of the Braille writing-assisted teaching system, sub-step 1.4 of the first embodiment employs an intra-frame spatial weighted fusion strategy to perform a single static weighted mean calculation on the effective depth set. That is, weights are assigned solely based on the planar Euclidean distance between each effective depth element and the marker center coordinates; pixels closer to the center are assigned higher confidence levels. Subsequently, a normalized weighted summation is performed on all depth elements to obtain the marker depth value. This strategy treats each frame's depth calculation as a completely independent isolated event, without introducing any inter-frame constraints to verify the temporal consistency of the output results.

[0027] However, in practical applications of writing aids for the blind, users continuously move the pen, and the true spatial depth of the markers on the pen inevitably exhibits a smooth, gradual change between frames under the constraints of physical laws. The depth change between adjacent frames is constrained by both the arm's movement speed and the frame rate, making sudden, unexplained changes impossible. Simultaneously, in practice, depth cameras often output transient anomalies that deviate significantly from the true depth in individual frames due to hand shadows obscuring the marker surface, directional reflections of infrared structured light by the marker coating, or thermal noise pulses from the sensor itself. The first embodiment lacks any ability to identify or suppress such single-frame depth abrupt changes; the anomaly, after spatial weighting, directly becomes the final depth output for that frame. This deficiency is amplified in the downstream interaction chain of this system: the marker depth value is directly converted into a real-time height difference via the height difference calculation formula in main step four, and this real-time height difference is then conditionally compared with a height fluctuation threshold in the pen-raising and lowering state determination in main step five. If a depth value in a frame experiences a sudden change of several centimeters due to transient noise, the system will incorrectly determine whether the pen tip has detached from or is in contact with the writing surface in that frame, and will then issue a false prompt to the blind user to lift or lower the pen. For blind and low-vision users who rely entirely on auditory feedback for writing, this false prompt not only interrupts the ongoing writing process but also undermines the user's trust in the accuracy of the system's guidance, severely reducing the efficiency and experience of independent writing training.

[0028] To address the aforementioned technical deficiencies, a second embodiment is proposed. More specifically, in another specific example of this application, Figure 5This is a flowchart of step 1.4 of the blind writing-assisted teaching method according to an embodiment of this application. Figure 5 As shown, step 1.4 includes: Step 1.4.1, based on the coordinates of each pixel and the center of the marker, performing intra-frame spatial weighting on the effective depth set to obtain intra-frame weighted depth values, and performing exponential smoothing prediction on the depth temporal sliding window to obtain temporal predicted depth values; Step 1.4.2, performing temporal deviation factor quantization evaluation on the intra-frame weighted depth values ​​and the temporal predicted depth values ​​to obtain a temporal deviation factor; Step 1.4.3, based on the temporal deviation factor, performing adaptive linear fusion on the intra-frame weighted depth values ​​and the temporal predicted depth values ​​to obtain marker depth values, and pushing the marker depth values ​​into the depth temporal sliding window to complete iterative updates.

[0029] In step 1.4.1, based on the coordinates of each pixel and the center of the marker, the effective depth set is spatially weighted within the frame to obtain the intra-frame weighted depth value, and the temporal sliding window of the depth is exponentially smoothed to obtain the temporal predicted depth value. It should be understood that when the depth camera suddenly obscures the structured light projection area or the marker surface due to a hand shadow in a certain frame, causing directional reflection, even if the elements in the effective depth set exhibit a systematic shift, the shifted data is still indiscriminately fused into the final depth output of the current frame. This abnormal depth value, after being transmitted to the pen-raising / lowering state determination via downstream height difference calculation, will trigger a false pen-raising / lowering prompt, causing interactive misleading for blind and low-vision users who rely entirely on auditory feedback. Therefore, in the technical solution of this application, based on the coordinates of each pixel and the center of the marker, the effective depth set is spatially weighted within the frame to obtain the intra-frame weighted depth value, and the temporal sliding window of the depth is exponentially smoothed to obtain the temporal predicted depth value. This simultaneously generates two depth estimates with different information sources: one from independent spatial domain observation of the current frame, and the other from temporal domain inertial prediction of the historical frame sequence. This provides a dual-channel data foundation for subsequent quantitative evaluation of the temporal deviation factor. In this way, while retaining the real-time observation capability of the current frame's spatial domain, a priori constraint signal based on the motion inertia of historical depth trajectories is introduced, enabling the system to obtain a basis for quantitatively evaluating the reliability of the current frame's depth measurement. This lays the foundation for a dual-channel comparison at the signal level for subsequent identification of transient noise pulses and implementation of gradient suppression.

[0030] In a specific example of this application, firstly, for the effective depth set of the current frame input, a spatially weighted mean operation is performed based on the Euclidean distance between the pixel position of each element and the coordinates of the marker center, to obtain an intra-frame weighted depth value that only reflects the local spatial information of the current single frame. This intra-frame spatial weighting operation is consistent with the weighted fusion strategy in the first embodiment, that is, each effective depth element in the effective depth set is assigned a confidence weight based on the planar Euclidean distance between its pixel coordinates and the coordinates of the marker center; pixels closer to the center are assigned higher weight values. Then, a normalized weighted summation operation is performed on all effective depth elements to obtain the intra-frame weighted depth value. The physical meaning of the intra-frame weighted depth value is: without considering any historical frame information, it is an independent estimation of the spatial depth of the marker center region based solely on the spatial distribution characteristics of each effective pixel within the current frame depth sampling area. This estimation value completely preserves the original observation information of the current frame depth sensor, including normal depth changes caused by the user's actual pen-raising and lowering actions, as well as possible transient noise interference.

[0031] Next, an exponentially weighted smoothing recursive operation is performed on the depth temporal sliding window continuously maintained by the system. Constrained by the motion inertia trend of historical depth trajectories, a priori estimate of the current frame's depth is generated, i.e., the temporal predicted depth value. The depth temporal sliding window is a first-in-first-out data structure continuously maintained by the system during operation. This window caches the confirmed marker depth output values ​​of the most recent consecutive frames. In this example, the window length is set to 10 frames, corresponding to a time span of approximately 333 milliseconds. The exponentially weighted smoothing recursive process is expressed as follows: in, This represents the temporal prediction depth value of the current frame (time t). This represents the confirmed depth output value for the most recent frame (time t-1) within the depth-time sliding window. This represents the temporal prediction depth value at the previous time step. This represents the smoothing coefficient, ranging from zero to one, used to control the balance of response sensitivity between historical inertia holding and latest observation following; in this example, it is set to 0.3. The first term on the right-hand side of the above formula... This term represents the contribution of the most recently confirmed depth output value within the depth time-series sliding window. It assigns a certain proportion of direct influence to the latest confirmed output, enabling the time-series predicted depth value to track the actual movement trend of the marker; the second term on the right side of the formula... This represents the contribution to the inertia of the previous time-series prediction value. This term continues the motion inertia of the historical prediction trajectory in a complementary proportion, preventing the time-series prediction depth value from over-responding to single-frame fluctuations. The two are connected by a smoothing coefficient. Perform linear matching. The larger the value, the faster the time series prediction depth follows the latest confirmed output. The smaller the value, the stronger the historical inertia is preserved. In this example... Setting it to 0.3 means that the temporal prediction depth value maintains the inertial trend of the historical prediction trajectory by 70% and only follows the change of the confirmation output of the most recent frame by 30%. This parameter configuration enables the temporal prediction depth value to smoothly track the slow and gradual depth changes in the user's normal writing motion, while having a significant inertial suppression effect on transient changes in a single frame.

[0032] In practical applications of writing aids for the blind, users continuously move the pen. The true spatial depth of the markers on the pen, constrained by physical laws, inevitably exhibits a smooth, gradual change between frames. The depth variation between adjacent frames is constrained by both the arm's movement speed and the frame rate, preventing sudden, unexplained changes. When the pen moves at a constant or low acceleration, the temporal predicted depth value closely approximates the true depth trajectory, providing a stable anchor point for subsequent anomaly detection. However, when the depth camera experiences signal distortion due to sudden occlusion of the structured light projection area by hand shadows or specular reflection from the marker surface, the intra-frame weighted depth value significantly deviates from this temporal anchor point, thus creating a quantifiable criterion space for anomaly identification. After the above dual-channel parallel computation, the system simultaneously holds two depth estimation results in the current frame: the intra-frame weighted depth value and the temporal predicted depth value. The former carries the complete spatial domain observation information of the current frame but may contain transient noise contamination, while the latter carries the temporal domain inertial prior of the historical frame sequence but has a lag response to real abrupt changes. The two form a complementary dual relationship in terms of information source and noise sensitivity, providing a dual benchmark input with clear physical meaning for the quantitative evaluation of the temporal deviation factor in the next sub-step.

[0033] In step 1.4.2, the intra-frame weighted depth value and the temporal predicted depth value are quantitatively evaluated using a temporal deviation factor to obtain the temporal deviation factor. It should be understood that since the intra-frame weighted depth value and the temporal predicted depth value generated synchronously in the previous sub-step come from the independent spatial domain observation of the current frame and the temporal domain inertial prediction of the historical frame sequence, respectively, they form a complementary dual relationship in terms of information sources. However, the system has not yet established a quantitative mechanism to measure the degree to which the measured result of the current frame deviates from the historical motion trajectory. If the two signals are directly mixed at a fixed ratio, it is impossible to distinguish whether the depth change of the current frame originates from the user's actual pen-raising and lowering actions or from transient noise interference from the depth camera. A continuous and controllable numerical metric is needed to provide an adjustment basis for the subsequent fusion weight allocation, rather than a simple binary threshold truncation. Therefore, in the technical solution of this application, the intra-frame weighted depth value and the temporal predicted depth value are further evaluated by temporal deviation factor quantification to obtain the temporal deviation factor. This transforms the deviation of the current frame's spatial domain observation value from the temporal domain inertial prediction value into a dimensionless continuous numerical index, providing precise control input for the dynamic weight allocation in subsequent adaptive linear fusion. In this way, by transforming the discrete binary judgment of normal and abnormal into a continuous numerical metric, the system gains the ability to respond to measurement disturbances of different degrees in a gradient manner. When the intra-frame measured depth and the temporal predicted depth are highly consistent, the output deviation factor approaches zero, indicating the reliability of the current frame. When the two deviate significantly, the output deviation factor increases significantly, indicating that the frame is suspected of noise contamination.

[0034] In a specific example of this application, firstly, the intra-weighted depth value and the temporally predicted depth value output from the previous sub-step are extracted, and the absolute deviation between them is calculated. This absolute deviation reflects the original numerical difference between the current frame's spatial domain observation result and the temporal domain inertial prediction result, and its dimension is the same as the depth measurement value, with the unit being millimeters. Next, to eliminate the scale difference effect of the deviation when the marker is at different absolute depth positions, the absolute deviation value is normalized by division using the temporally predicted depth value as a normalization benchmark, converting the dimensionless deviation value into a dimensionless relative deviation ratio. Then, the absolute deviation between the intra-weighted depth value and the temporally predicted depth value is divided by the temporally predicted depth value for normalization, resulting in a dimensionless temporal deviation factor, expressed as: in, Indicates the timing deviation factor of the current frame. This represents the intra-frame weighted depth value of the current frame. This represents the temporal predicted depth value of the current frame. This represents the smallest positive constant protected against division by zero; in this example, it is 0.001. (Numerator of the formula) This represents the absolute deviation between the intra-frame weighted depth value and the temporally predicted depth value. Physically, this term represents the original distance by which the current frame's independent spatial domain observation deviates from the historical temporally predicted trajectory. Taking the absolute value ensures that the deviation is represented as a positive number regardless of whether the intra-frame observation is higher or lower than the temporally predicted value, making the temporal deviation factor symmetrically sensitive to positive and negative anomalies. (The denominator of the formula is...) Using the predicted depth value as the normalization benchmark, its physical meaning is to convert the absolute deviation into a proportional deviation relative to the current expected depth level. This normalization process eliminates the scaling effect of the marker's position at different absolute depths (e.g., under two installation heights, 40 cm and 60 cm from the writing surface), ensuring that the temporal deviation factor has a uniform numerical meaning and consistent threshold applicability under different system installation configurations. The introduction of this feature prevents the division-to-zero anomaly caused by the time-series prediction depth value approaching zero before a valid historical sequence has been established during the system initialization phase.

[0035] In practical applications of writing assistance for the blind, when the pen moves normally and smoothly near the writing surface, the user's arm movement speed and the 30 frames per second acquisition rate together constrain the depth change between adjacent frames to no more than a few millimeters. The intra-frame weighted depth value is highly consistent with the temporal predicted depth value, and the temporal deviation factor approaches zero, indicating that the depth sensor in the current frame is working within a reliable range. Subsequent fusion should prioritize the intra-frame measured value to maintain an immediate response to the user's actual actions. When the depth camera's infrared echo signal is distorted due to the sudden obscuring of the structured light projection area by the hand shadow or specular reflection from the marker surface, the intra-frame weighted depth value will deviate sharply by several centimeters in that frame. The intra-frame measured depth will deviate sharply from the temporal predicted trajectory, and the temporal deviation factor will increase significantly, accurately indicating that the frame is suspected of noise pollution. Subsequent fusion should prioritize the temporal predicted value to suppress the abnormal measurement output of that frame. By transforming the discrete binary judgment of normal and abnormal into a continuous numerical metric, the system gains the ability to respond to measurement disturbances of different degrees in a gradient manner, avoiding the judgment oscillation problem caused by the hard threshold truncation method in the critical region, and providing a continuous and smooth weight control input for the adaptive linear fusion driven by the time deviation factor in the next sub-step.

[0036] In step 1.4.3, based on the temporal deviation factor, the intra-frame weighted depth value and the temporal predicted depth value are adaptively linearly fused to obtain the marker depth value, and the marker depth value is then pushed into the depth temporal sliding window to complete the iterative update. It should be understood that since the temporal deviation factor output in the previous sub-step has quantified the deviation of the current frame's spatial domain observation value from the temporal domain inertial prediction value into a continuous numerical index, but this index itself has not yet been converted into a specific mixing ratio of the intra-frame weighted depth value and the temporal predicted depth value, the system still lacks an execution mechanism to implement the deviation judgment into a fusion action. Furthermore, after the current frame completes the depth calculation, its result needs to be fed back to the historical sequence to support the temporal prediction recursion of subsequent frames; otherwise, the data in the depth temporal sliding window will expire over time and fail to reflect the latest motion state of the marker. Therefore, in the technical solution of this application, based on the temporal deviation factor, the intra-frame weighted depth value and the temporal predicted depth value are adaptively linearly fused to obtain the marker depth value. The marker depth value is then pushed into the depth temporal sliding window for iterative updates. This transforms the reliability of the current frame observation value quantified by the temporal deviation factor into a dynamic mixing ratio of the two depth estimates, achieving an adaptive balance between preserving the real-time tracking sensitivity of normal motion frames and suppressing transient pulses in abnormal noise frames. Simultaneously, the closed-loop iteration of the sliding window provides historical data support containing the latest confirmed values ​​for the temporal prediction recursion of subsequent frames. In this way, the marker depth value can quickly follow the depth change when the user actually lifts and puts down the pen. When the depth camera outputs transient noise, it automatically suppresses abnormal measurement outputs with the inertia of historical trajectories. The cross-frame cyclic data closed loop ensures that the temporal prediction mechanism is always based on the latest confirmed depth sequence during long-term system operation, thereby effectively eliminating false pen-lifting and-falling prompts caused by single-frame depth mutations and ensuring the temporal consistency and physical rationality of the voice guidance signals received by blind and low-vision users during independent writing training.

[0037] In a specific example of this application, firstly, the timing trust weight is calculated based on the ratio of the timing deviation factor to a preset deviation switching threshold, expressed as: in, This represents the time-series trust weight, and its value ranges from zero to one. Indicates the timing deviation factor of the current frame. This represents a preset deviation switching threshold constant, used to calibrate the boundary between the depth fluctuation amplitude of normal writing motion and transient noise pulses; in this example, its value is 0.05. The ratio part inside the formula... The timing deviation factor is linearly normalized relative to the deviation switching threshold. When the timing deviation factor is within the range of zero to the deviation switching threshold, this ratio changes linearly between zero and one, achieving a gradient response to different degrees of deviation; the outer minimum value operation... The comparison results are truncated at an upper bound to ensure that the temporal trust weight saturates to one when the time-series deviation factor exceeds the deviation switching threshold, preventing it from increasing indefinitely and thus avoiding weight overflow that could disrupt the numerical stability of the fusion operation. In practical applications of blind writing assistance, when the time-series deviation factor is much smaller than the deviation switching threshold, the temporal trust weight approaches zero, and the fusion result is dominated by the intra-frame measured value, ensuring the system has an instantaneous response capability to depth changes in the user's actual pen-lifting and pen-falling actions. When the time-series deviation factor approaches or exceeds the deviation switching threshold, the temporal trust weight approaches 1, and the fusion result is dominated by the temporal prediction value, automatically suppressing abnormal measurement outputs in the current frame by utilizing the continuity and inertia of historical trajectories.

[0038] Next, an adaptive linear fusion operation is performed on the two depth estimates based on temporal trust weights to obtain the final marker depth value, expressed as: in, This represents the final output marker depth value. Represents the time-series trust weight. Indicates the intra-frame weighted depth value. This represents the temporal prediction depth. The above fusion formula uses a complementary linear interpolation structure with temporal trust weights. This serves as a dynamic modulation factor, simultaneously controlling the contribution ratio of both signals. The first term on the right-hand side of the formula... Indicates the intra-frame weighted depth value after complementary weighting The modulated contribution, when the time-order trust weight approaches zero, has a contribution ratio approaching one, meaning the system fully trusts the spatial domain independent observations of the current frame. This corresponds to the state where the pen is making normal, stable movements and the depth sensor is operating within the confidence interval, giving the system the ability to track the user's real actions instantly without delay; the second term on the right side of the formula... This represents the time series prediction depth value weighted by the time series trust weight. The modulated contribution, when the sequential trust weight approaches one, has a contribution ratio approaching one, meaning the system completely relies on the inertial prediction value of historical depth trajectories to replace the abnormal measurement output of the current frame. This situation corresponds to the state where the depth of the current frame deviates significantly from the true trajectory due to hand shadows or specular reflections. The system automatically filters transient noise pulses using historical motion inertia. The sum of the two items always equals one order of magnitude of depth, ensuring the correct physical dimensions of the fusion result. In this example, when the user's arm moves the pen at a normal speed, the inter-frame depth change is approximately 1 to 3 millimeters, the temporal deviation factor is approximately 0.002 to 0.006, which is much smaller than the deviation switching threshold of 0.05, and the temporal trust weight is approximately 0.04 to 0.12. The fusion result retains the intra-frame measured values ​​at a rate of 88% to 96%, ensuring sensitive tracking of real motion. However, when the depth camera outputs transient anomalies on the order of 2 to 5 centimeters, the temporal deviation factor jumps to 0.04 to 0.1, the temporal trust weight saturates at 0.8 to 1.0, and the fusion result uses the temporal predicted values ​​at a rate of 80% to 100%, compressing the penetration amplitude of the abnormal pulse to a tolerable range not exceeding the height fluctuation threshold.

[0039] Then, after the fusion operation is completed, the confirmed marker depth values ​​of the current frame are pushed into the tail of the depth temporal sliding window, and outdated data at the head of the queue is popped. This completes the first-in-first-out iterative update of the window, enabling the next frame to perform temporal prediction recursion based on the historical sequence containing the latest confirmed values, forming a data loop across frames. This iterative update mechanism ensures that the depth temporal sliding window always maintains the confirmed marker depth output values ​​of the most recent consecutive frames, allowing the historical data on which the exponential smoothing prediction recursion depends to continuously reflect the latest motion state of the markers, avoiding cumulative offsets between the temporal predicted depth values ​​and the actual trajectories due to outdated window data.

[0040] In summary, by introducing inter-frame depth temporal smoothing as a priori constraint and replacing the original single-frame static weighted output strategy with an adaptive fusion mechanism driven by temporal deviation factors, this second embodiment enables the depth calculation stage to maintain real-time tracking sensitivity to the normal continuous movement of the writing pen while achieving autonomous identification and gradient suppression capabilities for transient noise pulses from the depth camera. The inter-frame stability of marker depth values ​​is significantly improved. After being transmitted to the pen-starting / stopping state determination stage via downstream height difference calculation, it effectively eliminates false pen-starting / stopping prompts caused by single-frame depth mutations, ensuring the temporal consistency and physical rationality of the voice guidance signals received by blind and low-vision users during independent writing training, reducing the frequency of interactive interference, and improving the reliability and user trust of the assisted teaching system.

[0041] Specifically, in step two, based on the boundary points of the writing area, normalized coordinate mapping and contact height calibration are performed on the marker center coordinates and marker depth values ​​to obtain virtual space mapping parameters, marker calibration height, and height fluctuation threshold. It should be understood that since the marker center coordinates output in step one are in the pixel coordinate system of the original depth camera image, the scale and range of this coordinate system depend entirely on hardware parameters such as camera resolution, installation height, and lens focal length. Under different installation configurations, the same physical writing displacement will correspond to different orders of magnitude of pixel coordinate changes, causing subsequent orientation angle calculations and distance determination to be unable to be performed using a unified threshold standard. Simultaneously, the marker depth value is the absolute spatial distance from the camera imaging plane to the marker center area. This absolute distance value shifts overall with changes in camera installation height and cannot be directly used to determine whether the pen tip is in contact with the writing surface. The system needs to establish a unified coordinate representation decoupled from the hardware configuration and a relative height reference decoupled from the absolute depth. Therefore, in the technical solution of this application, based on the boundary points of the writing area, the coordinates of the marker center and the marker depth value are further normalized and the contact height is calibrated to obtain virtual space mapping parameters, marker calibration height, and height fluctuation threshold. This normalizes the physical pixel coordinate system into a virtual coordinate space independent of hardware configuration, and transforms the absolute depth measurement value into a height reference and tolerance criterion relative to the contact state of the writing surface. In this way, the system can complete the direction calculation and distance determination in a unified virtual coordinate domain under different paper sizes, different camera installation heights, and different user writing ranges. It also uses an adaptive height fluctuation threshold to accommodate vertical deviations caused by different pen tip lengths, marker installation positions, and paper thicknesses, providing a hardware-independent and scene-adaptive parameter basis for subsequent frame-by-frame virtual mapping, direction angle calculation, and pen lifting and lowering state determination.

[0042] More specifically, in the embodiments of this application, step two includes: step 2.1, in response to the operation of the writing pen being placed sequentially at the extreme boundary position of the writing area during the calibration phase, sampling and caching the center coordinates of markers in multiple consecutive frames to construct a planar boundary coordinate set, and simultaneously caching the height mapping values ​​of markers in the corresponding frames to construct a contact height sample set; step 2.2, based on preset virtual resolution parameters, constructing virtual space mapping parameters by performing affine transformation matrices on the steady-state coordinates of the upper left and lower right corners of the planar boundary coordinate set; step 2.3, performing baseline height estimation and tolerance adaptation on all sample values ​​in the contact height sample set to obtain a height fluctuation threshold.

[0043] Specifically, in a specific example of this application, firstly, in response to the operation of placing the writing pen sequentially at the extreme boundary positions of the writing area during the calibration phase, the system guides the user through voice prompts to place the writing pen at the upper left corner of the writing area and keep it in a posture with the pen tip in contact with the writing surface and the pen barrel approximately perpendicular to the writing surface for about 2 seconds. Then, it is placed at the lower right corner and kept in the same posture for about 2 seconds. Within the static time window of each calibration position, the system continuously collects about 60 frames of data at a frame rate of 30 frames per second. The median of the marker center coordinates of multiple consecutive frames at each position is used for steady-state filtering to eliminate the influence of hand tremors. The steady-state coordinate values ​​of the two calibration positions are cached to construct a planar boundary coordinate set. This set contains four values: the pixel horizontal and vertical coordinates of the upper left and lower right reference points of the writing area. At the same time, the marker depth values ​​of the corresponding frames of the two calibration positions are height-mapped and cached to construct a contact height sample set. This set contains the depth measurement values ​​of the markers when the pen tip contacts the writing surface at the two calibration positions. Next, based on the preset virtual resolution parameters, an affine transformation matrix is ​​constructed on the steady-state coordinates of the upper left and lower right corners of the plane boundary coordinate set to obtain the virtual space mapping parameters. In this example, the resolution of the virtual coordinate space is set to 100×100, and the mapping relationship is expressed as follows: in, and These represent the horizontal and vertical coordinates of the marker in the virtual coordinate space, respectively. and These represent the maximum horizontal resolution width and maximum vertical resolution height of the preset virtual resolution parameters, respectively, both of which are set to 100 in this example; and These represent the pixel x and y coordinates of the top-left reference point in the plane boundary coordinate set, respectively. and Let (x, y) represent the pixel x and y coordinates of the lower right corner reference point in the plane boundary coordinate set, respectively; and let (x, y) represent the pixel x and y coordinates of the center coordinates of the marker to be mapped. The fractional part of the first expression in the above formula... The horizontal scaling factor physically represents the ratio between the total horizontal span of the virtual coordinate space and the actual horizontal pixel span of the writing area in the image. It normalizes the pixel width of the writing area to within the standard width range of the virtual space, and then multiplies it by... The first expression represents the horizontal offset of the current coordinates relative to the top-left reference point, completing the translation and scaling transformation from the absolute pixel position to the relative virtual position. The second expression has a similar structure, implementing a normalized mapping of the vertical dimension. This mapping relationship is encapsulated as virtual space mapping parameters and stored in system memory for use in subsequent steps three (target keypoint sequence mapping) and four (frame-by-frame real-time coordinate mapping).

[0044] Furthermore, when rotation or perspective distortion in the writing area results in a non-orthogonal geometric relationship between the writing surface and the camera imaging plane, the aforementioned linear scaling mapping based on two boundary points will introduce geometric distortion errors. In this case, the virtual coordinate space calibration module uses four calibration points to establish a homography transformation matrix, uniformly mapping the acquired coordinates to the virtual coordinate space. When homography transformation is used, the mapping relationship is as follows: in, Represents the homography transformation matrix; This represents the scaling factor of homogeneous coordinates; and This represents the x and y coordinates of the marker's center in the pixel coordinate system of the acquired image; and This represents the horizontal and vertical coordinates of the marker after it has been mapped to the virtual coordinate space. The homography transformation matrix is ​​a 3×3 matrix with 8 degrees of freedom. It is uniquely and deterministically solved using 8 constraint equations provided by four calibration points. This compensates for geometric deformations such as writing surface tilt, non-aligned camera mounting, or paper warping, ensuring that the virtual space mapping parameters maintain mapping accuracy even under non-ideal mounting conditions. In the preferred embodiment of this example, the optical axis of the depth camera lens points perpendicularly to the writing surface, the writing area is a regular rectangle, and two-point linear scaling mapping already meets the accuracy requirements. When the system detects that the ratio of the four corner coordinates deviates from the rectangular constraint by more than a preset angle threshold during the calibration phase, it automatically switches to the aforementioned homography transformation mode.

[0045] Then, a baseline height estimate is performed on all sample values ​​in the contact height sample set. The arithmetic mean of the depth measurements at two calibration locations in the contact height sample set is then calculated to obtain the marker calibration height, which characterizes the standard height of the marker when the pen tip contacts the writing surface. This marker calibration height is expressed as: in, Indicates the calibration height of the marker; This represents the total number of samples in the contact height sample set, which is 2 in this example; This represents the marker depth measurement value at the i-th calibration position in the contact height sample set. Further, based on the dispersion of the contact height sample set and a preset tolerance parameter, a tolerance-adaptive process is performed to obtain a height fluctuation threshold. This threshold accommodates vertical deviations caused by different pen tip lengths, marker installation positions, paper thicknesses, and user operation errors. In this example, the height fluctuation threshold is fixed at 2 cm. That is, when the difference between the marker depth value and the marker calibration height does not exceed 2 cm, the system determines that the pen tip is in contact with the writing surface; when the difference exceeds 2 cm, the system determines that the pen tip has detached from the writing surface. This threshold setting comprehensively considers the depth measurement noise level of approximately 1 to 2 mm at a working distance of 40 to 60 cm by the depth camera, as well as the natural fluctuation range of the marker ring height caused by fine adjustments in the pen grip angle during normal writing. This ensures that no false judgments are triggered during normal writing actions, while also having sensitive detection capabilities for explicit pen-lifting actions by the user.

[0046] Specifically, in step three, key points of the content to be written are extracted and resampled in an orderly manner, and mapped to a virtual coordinate space using virtual space mapping parameters to obtain a sequence of target key points. It should be understood that since the blind writing assistance teaching system needs to provide users with orientation angles and distance information for each key point during the formal writing guidance stage, and the content to be written (such as Chinese characters, letters, or numbers) is stored in the system in character encoding form, it cannot directly participate in spatial domain orientation calculations and distance comparisons. The system needs to transform the abstract character content into an ordered set of coordinate points with clear spatial coordinates and stroke order relationships in the virtual coordinate space. Furthermore, the coordinate domain of this set of coordinate points must be within the same coordinate system as the virtual plane coordinates obtained by mapping the center coordinates of the marker in step four using virtual space mapping parameters, in order to achieve frame-by-frame coordinate subtraction and orientation angle calculations. Therefore, in the technical solution of this application, key points of the content to be written are further extracted and resampled in an orderly manner, and mapped to a virtual coordinate space using virtual space mapping parameters to obtain a sequence of target key points. This transforms the content to be written from abstract character encoding into an ordered set of spatial navigation points in the virtual coordinate space, which has stroke order, intra-stroke order, and unified coordinate representation. This allows the system to directly extract the virtual coordinates of the currently active key points from the target key point sequence point by point during the formal writing guidance stage, and perform difference calculations with the real-time virtual plane coordinates of the markers. This ensures that the direction angle calculation and distance determination are performed within a unified virtual coordinate domain, eliminating direction offsets or distance scale distortions caused by inconsistencies in the coordinate system.

[0047] More specifically, in the embodiments of this application, step three includes: step 3.1, converting the content to be written into a reference character image through a font rendering engine, and performing skeleton refinement processing on the reference character image to obtain a character skeleton image; step 3.2, performing basic sequence resampling on the character skeleton image to obtain an original pixel key point sequence; step 3.3, based on the affine transformation matrix in the virtual space mapping parameters, performing target coordinate system mapping calculation on each key point in the original pixel key point sequence to obtain a target key point sequence.

[0048] Specifically, in one particular example of this application, the content to be written is a single Chinese character selected by the user through the system interface or voice command. First, the content to be written is converted into a reference character image by the font rendering engine, and the reference character image is then subjected to skeleton thinning processing. Specifically, the system calls the font rendering engine to render the character encoding of the content to be written into a reference character image with a resolution of 256×256 pixels using a preset font (such as KaiTi). This image is a binary grayscale image with a white background and black characters. Subsequently, the Zhang-Suen skeleton thinning algorithm is applied to the reference character image for iterative erosion processing, gradually removing the outer pixels of the stroke outline until only a single pixel width of the centerline skeleton is retained for each stroke segment, resulting in a character skeleton image that retains the original stroke topological connectivity. Next, the character skeleton image is resampled to obtain the original pixel keypoint sequence. Specifically, connected component analysis and stroke library matching are performed on the character skeleton image. The skeleton image is then divided into a set of independent stroke segments with a specific stroke order according to the standard writing stroke order. For each stroke segment, start and end point detection, intersection point detection, inflection point detection, and equidistant interpolation are performed sequentially to extract basic keypoints, resulting in a complete set of basic keypoints for the entire character. Subsequently, the basic keypoints of each stroke segment are resampled with density adjustment. For the first... Each stroke has a number of basic key points. The density adjustment parameter is The number of keypoints after resampling is ,but: in, This represents the number of key points after resampling the j-th stroke. This represents the number of fundamental key points of the j-th stroke. This represents the density adjustment parameter, which is a non-negative adjustable parameter that can be adjusted based on stroke length, stroke curvature, target content complexity, or user proficiency. In this example, it is fixed at a value of 1. This indicates the floor function (rounding up). In the formula above... Some are amplified based on the number of basic key points according to the proportion specified by the density adjustment parameters. When the value is 1, the number of keypoints after resampling is twice the number of basic keypoints, reducing the keypoint spacing to half of the original spacing. This provides denser navigation anchors for blind users, reducing the difficulty of directional adjustment between adjacent keypoints. The rounding up operation ensures that each stroke segment retains at least an integer number of keypoints as the result of the basic keypoint amplification. After resampling all stroke segments, the original pixel keypoint sequence is obtained by concatenating the stroke order with the keypoint order within the stroke. Each keypoint in this sequence is represented by pixel coordinates in the character skeleton image.

[0049] Then, based on the affine transformation matrix in the virtual space mapping parameters, the target coordinate system mapping is calculated for each key point in the original pixel key point sequence to obtain the target key point sequence. Since the coordinates of each key point in the original pixel key point sequence are in the 256×256 pixel character skeleton image coordinate system, they need to be mapped to the virtual coordinate space corresponding to the writing area. The system uses the affine transformation matrix encapsulated in the virtual space mapping parameters to perform homogeneous coordinate multiplication on the two-dimensional pixel coordinates of each key point in the original pixel key point sequence, projecting them from the character image pixel domain to the 100×100 virtual coordinate space domain. During the mapping process, the coordinate offset is adjusted accordingly based on the preset arrangement position of the content to be written in the writing area (such as center placement) to ensure that the virtual coordinates of each key point in the target key point sequence fall within the effective range of the writing area. The target key point sequence obtained after the above mapping calculation is an ordered set of all target key points expressed in virtual coordinate space, which has stroke order and stroke sequence. Each target key point contains three attribute information: virtual horizontal coordinate, virtual vertical coordinate and the stroke number to which it belongs. This sequence is stored in the system memory for use in both the frame-by-frame direction angle calculation in step four and the key point iterative switching in step five.

[0050] Specifically, in step four, the marker center coordinates are mapped frame-by-frame using virtual space mapping parameters to obtain the mapping result. The mapping result is then subtracted from the current key point in the target key point sequence, and the orientation angle and distance are calculated. Simultaneously, the height difference is quantized based on the marker depth value and the marker calibration height to obtain the planar orientation angle, arrival distance, and real-time height difference. It should be understood that since the marker center coordinates output frame-by-frame in step one are in the pixel coordinate system of the original depth camera image, while the key points in the target key point sequence generated in step three have been mapped to the virtual coordinate space, the two belong to different coordinate systems and have inconsistent dimensions and scales. Therefore, it is impossible to directly perform coordinate difference calculations to obtain orientation and distance information. Furthermore, the marker depth value is an absolute spatial distance rather than a height offset relative to the writing surface, making it impossible to directly compare with the height floating threshold to determine the pen tip contact state. The system needs to unify the real-time position information of the markers into the virtual coordinate space and simultaneously complete the relative situation calculation in both the horizontal and vertical dimensions. Therefore, in the technical solution of this application, the virtual space mapping parameters are further used to map the center coordinates of the marker frame by frame to obtain the mapping result. The mapping result is then subtracted from the current key point in the target key point sequence, and the orientation angle and distance are calculated. Simultaneously, the height difference is quantized based on the marker depth value and the marker calibration height to obtain the planar orientation angle, arrival distance, and real-time height difference. This transforms the original sensor data of the marker into three-dimensional relative situational parameters that can be directly used for interactive decision-making in each acquisition frame. In this way, after the system completes the real-time projection from the original pixel coordinate domain to the virtual coordinate domain in each frame, it can simultaneously obtain the yaw azimuth information required for the current position of the writing pen to point to the next target key point in the horizontal plane, the planar Euclidean distance information between the two, and the vertical deviation information of the pen tip relative to the writing surface. This provides complete three-dimensional decision input for the direction prompt generation, arrival determination, and pen lifting and lowering state determination in step five.

[0051] Figure 6 This is a flowchart of step four of the blind writing-assisted teaching method according to an embodiment of this application. Figure 6 As shown, step four includes: Step 4.1, using the affine transformation matrix in the virtual space mapping parameters, performing real-time virtual plane mapping on the center coordinates of the marker in the current frame to obtain the virtual plane coordinates of the marker; Step 4.2, extracting the coordinates of the currently active key point from the target key point sequence, and performing planar relative situation calculation on the virtual plane coordinates of the marker and the coordinates of the currently active key point to obtain the planar orientation angle and the arrival distance; Step 4.3, calculating the algebraic difference between the depth value of the marker in the current frame and the calibration height of the marker to obtain the real-time height difference.

[0052] More specifically, in a concrete example of this application, firstly, the affine transformation matrix in the virtual space mapping parameters is used to perform real-time virtual plane mapping on the marker center coordinates of the current frame. Specifically, the virtual space mapping parameters generated and stored in the system memory during the calibration stage in step two are extracted. The pixel x and y coordinates of the marker center coordinates output in step one of the current frame are substituted into the affine transformation mapping relationship to calculate the horizontal and vertical coordinates of the marker in the virtual coordinate space, which are then combined into the marker virtual plane coordinates. Next, the coordinates of the currently active keypoint are extracted from the target keypoint sequence. The relative situational calculation between the marker virtual plane coordinates and the currently active keypoint coordinates is performed to obtain the plane orientation angle and the reach distance. Specifically, the current marker virtual plane coordinates are taken as the current point, and the virtual coordinates of the currently active keypoint pointed to by the active pointer maintained in step five in the target keypoint sequence are taken as the target point. The plane coordinate difference vector from the current point to the target point is calculated. The plane coordinate difference vector is expressed as: in, This represents the planar coordinate difference vector at acquisition time t; and These represent the virtual x-coordinate and virtual y-coordinate of the currently active keypoint in the target keypoint sequence, respectively. and These represent the horizontal and vertical coordinates of the virtual plane coordinates of the marker in the current frame, respectively. The plane orientation angle and distance to reach are calculated based on the plane coordinate difference vector, and are expressed as follows: in, The plane orientation angle represents the acquisition time t, and its value ranges from -π to π. This represents the distance reached at acquisition time t, i.e., the planar Euclidean distance from the current point to the target key point, expressed in virtual coordinate units. In the above direction angle formula... The function uses the vertical coordinate difference The first parameter is the difference in the horizontal coordinates. The second parameter uniquely determines the yaw azimuth angle required to move from the current pen position to the next target key point within the complete four quadrants. Physically, this means the blind user needs to move the pen along this azimuth angle to approach the next target key point. The arrival distance formula obtains the linear spatial span between the horizontal and vertical components of the coordinate difference vector by taking the square root of the sum of the squares. Physically, this represents the planar distance that the pen's current position needs to traverse in the virtual coordinate space between the target key point and the target. This distance decreases monotonically as the pen approaches the target key point until the arrival threshold condition in step five is met, triggering key point switching. Then, the algebraic difference between the current frame marker depth value and the marker calibration height is calculated to obtain the real-time height difference, expressed as: in, This represents the real-time height difference at time t. This represents the depth value of the marker in the current frame; This represents the marker calibration height determined in step two, the calibration phase. The formula above subtracts the marker calibration height from the current frame marker depth value, and the resulting real-time height difference reflects the vertical offset of the current writing pen marker relative to the standard height when the pen tip contacts the writing surface. Under the coordinate convention of setting the direction away from the writing surface as the positive height direction in this system, a positive real-time height difference indicates that the pen tip has left the writing surface (in the pen-lifting state), and a real-time height difference approaching zero or being negative indicates that the pen tip is in contact with or touching the writing surface (in the pen-falling state). This value will be conditionally compared with the height floating threshold in step five to perform the pen-lifting and falling state determination. After the above three-way parallel calculation, the current frame simultaneously outputs three relative situational parameters: planar orientation angle, arrival distance, and real-time height difference. These parameters completely characterize the spatial relationship between the current position of the writing pen and the target key point and the writing surface from three dimensions: horizontal orientation, horizontal distance, and vertical height. These three parameters serve as complete input data for the interactive prompt generation and state machine iterative control in step five.

[0053] Specifically, in step five, the planar direction angle is quantified and throttling is stabilized based on the directional interval division to generate planar movement prompts. The real-time height difference is used to determine the pen-raising and lowering status based on the arrival distance and height fluctuation thresholds to generate pen-raising and lowering prompts. When the arrival distance meets the arrival threshold, the current key point is switched to a subsequent key point in the target key point sequence until all key points have been reached. It should be understood that since the planar direction angle, arrival distance, and real-time height difference output frame by frame in step four are all continuous numerical geometric parameters, blind and low-vision users cannot directly perceive and understand continuously changing angle and distance values. The system needs to convert these abstract numerical parameters into discrete interactive guidance signals that can be conveyed through the auditory channel. Simultaneously, it needs to control the signal broadcast frequency to avoid information overload and cognitive fatigue caused by continuous frame-by-frame broadcasting. Furthermore, it needs to coordinate the judgment logic in both horizontal navigation and vertical pen-raising and lowering dimensions and drive the orderly progressive switching of target key points to achieve a closed-loop guidance for complete strokes and even entire characters. Therefore, in the technical solution of this application, the planar direction angle is further quantified and throttling is stabilized based on the division of direction intervals to generate planar movement prompts. The pen-raising and lowering state is determined based on the real-time height difference according to the arrival distance and height fluctuation threshold to generate pen-raising and lowering prompts. When the arrival distance meets the arrival threshold, the current key point is switched to a subsequent key point in the target key point sequence until all key points are reached. This transforms continuous geometric situational parameters into discretized interactive guidance signals that are audible, understandable, and followable for blind users. A state machine-driven key point iteration mechanism realizes the complete guidance process from single-point navigation to full-character writing. In this way, blind users can learn, through voice prompts, which direction to move the pen, whether the pen tip has reached the target position, and whether to press the pen tip down to contact the writing surface or lift the pen tip to prepare for the next stroke. The invisible writing trajectory is transformed into an audible and followable step-by-step interactive process, enabling independent writing training without visual feedback or continuous assistance from others.

[0054] More specifically, in this embodiment, step five includes: step 5.1, based on a preset eight-directional equally divided sector interval, performing interval quantization index mapping on the planar direction angle, and generating a planar movement prompt after time window throttling logic and continuous multi-frame stability determination; step 5.2, when the arrival distance enters the preset arrival threshold range, performing a conditional comparison between the real-time height difference and the height fluctuation threshold to perform pen tip contact state determination, and generating pen lifting and lowering prompts; step 5.3, based on the comparison result between the arrival distance and the preset arrival threshold, incrementally iteratively updating the active pointer in the target key point sequence to switch to the subsequent key point, and when the active pointer crosses the boundary, determining that all key points have been reached and outputting a writing completion signal, terminating the current round of interactive guidance loop.

[0055] Specifically, in a specific example of this application, the system continuously executes the interactive judgment and output of this step in a frame-by-frame loop during the user's formal writing phase. First, based on a preset eight-directional equally divided sector interval, the planar direction angle is quantized and indexed for interval mapping. After time window throttling logic and continuous multi-frame stability judgment, a planar movement prompt is generated. Specifically, a complete range of planar direction angles is divided into eight direction intervals, each with an angle span of 45 degrees, corresponding to the eight standard compass directions: front, right front, right, right rear, rear, left rear, left, and left front. Each directional interval is represented as: in, Indicates the first The angle range of each direction interval; This indicates the number of directional divisions, which is fixed at 8 in this example; The index represents the direction interval, ranging from 1 to 8. In the formula above, each interval is... Expand half an interval angle to each side of the center. The range constitutes a left-closed, right-open interval, seamlessly and non-overlappingly dividing the complete value range of the planar direction angle into 8 equally wide sector intervals, ensuring that any planar direction angle value can be uniquely mapped to a certain standard direction. When the planar direction angle of the current frame enters the k-th direction interval, the system extracts the direction semantic label (such as forward, left, right backward, etc.) corresponding to that interval as a candidate movement prompt. Subsequently, the candidate movement prompts undergo dual filtering of broadcast throttling and direction stabilization: the broadcast throttling unit controls the frequency of repeated prompts with a minimum broadcast period of 0.5 seconds, a minimum angle change threshold of 22.5 degrees, and a minimum distance change threshold of 2 virtual coordinate units. When the time since the last broadcast is less than 0.5 seconds or the change in direction angle does not exceed 22.5 degrees and the change in distance does not exceed 2 virtual coordinate units, the prompt output of the current frame is suppressed; the direction stabilization unit requires that the direction interval index of 3 consecutive frames remain consistent before the actual broadcast of the current direction prompt is triggered, in order to eliminate the problem of frequent back-and-forth switching of direction intervals at the boundaries of adjacent intervals caused by the inter-frame micro-judder of the marker center coordinates. The planar movement prompts, after double filtering, are broadcast to the user in plain voice. Next, when the distance reaches a preset arrival threshold, a conditional comparison is performed between the real-time height difference and the height fluctuation threshold to determine the pen tip contact state and generate a pen-starting / falling prompt. Specifically, the pen-starting / falling prompt module prioritizes pen-starting positioning when the user approaches the starting point of each stroke. When the distance exceeds the preset arrival threshold, the system only outputs planar movement prompts to guide the user to approach the target point horizontally. The system further considers the following arrival conditions: in, Indicates the distance reached in the current frame; This represents a preset arrival threshold, which is fixed at 2 virtual coordinate units in this example. When the above conditions are met and the real-time height difference is greater than the height fluctuation threshold, the system determines that the pen tip has not yet touched the writing surface and outputs a pen-down prompt voice signal to guide the user to press the pen tip towards the writing surface; when the arrival distance is not greater than the preset arrival threshold and the real-time height difference is not greater than the height fluctuation threshold, the system determines that the pen tip has touched the writing surface and outputs a contact confirmation signal and enters the planar movement broadcast state of the current stroke; during the writing of the current stroke, the system prioritizes outputting planar direction prompts and does not continuously broadcast height prompts to reduce information density and avoid cognitive overload; when the last key point of the current stroke is determined to be reached, the system outputs a pen-lifting prompt voice signal to guide the user to lift the pen tip away from the writing surface until the real-time height difference exceeds the height fluctuation threshold to confirm the completion of pen lifting. Then, based on the comparison result of the arrival distance and the preset arrival threshold, the activity pointer in the target key point sequence is incrementally and iteratively updated to switch to the subsequent key point. Specifically, when the arrival distance meets the threshold, the system automatically updates the activity pointer in the target key point sequence to switch to the subsequent key point. When the arrival condition is met, the system outputs an arrival prompt tone signal and increments the active pointer of the target key point sequence maintained internally by one position. This allows step four to extract the updated coordinates of the next key point from the target key point sequence in subsequent iteration frames for planar relative situation calculation. The planar coordinate difference vector, planar direction angle, and arrival distance are then recalculated to initiate navigation guidance for the next target key point. When the active pointer goes out of bounds (i.e., the pointer value exceeds the total number of key points in the target key point sequence), the system determines that all key points have been reached, outputs a writing completion prompt tone signal, terminates the current round of interactive guidance loop, and completes the full writing guidance process for the current content to be written, from the starting key point of the first stroke to the ending key point of the last stroke. Based on the user's voice command, the system decides whether to proceed to the next round of repeated practice or switch to new content to be written.

[0056] Figure 7 This is a schematic diagram illustrating the directional sound path and stroke guidance of the blind writing assistance teaching method according to an embodiment of this application. Figure 7As shown, this is an example of a complete three-stroke writing guidance process with the Chinese character "大" (da) as the content to be written. The first stroke in the figure corresponds to the writing guidance for the horizontal stroke of the Chinese character "大". According to the arrangement direction of each key point of this stroke segment in the target key point sequence, the system outputs plane movement prompts to the user key point by key point. At the start of the stroke, a pen-down prompt is output to guide the user to bring the pen tip into contact with the writing surface, and then voice prompts for plane movement in the right direction are sequentially output to guide the user to push the writing pen rightward along the horizontal direction. The direction angles between adjacent key points all fall into the right direction interval after being quantized by the eight-direction interval. After it is determined that the end key point of the horizontal stroke is reached, the system outputs a pen-lift prompt to guide the user to lift the pen tip to prepare for the next stroke. The second stroke corresponds to the writing guidance for the left-falling stroke of the Chinese character "大". The system first outputs a plane movement prompt to guide the user to move the writing pen to the starting key point position of the left-falling stroke, outputs a pen-down prompt after arrival, and then sequentially outputs voice prompts for plane movement in the rear-left direction to guide the user to push the pen along the lower-left direction. This direction angle falls into the rear-left direction interval after quantization, and a pen-lift prompt is output after the end key point of the left-falling stroke is reached. The third stroke corresponds to the writing guidance for the right-falling stroke of the Chinese character "大". The system guides the user to move to the starting key point of the right-falling stroke and then outputs a pen-down prompt. Then, according to the direction change of the right-falling stroke trajectory, voice prompts for plane movement in directions such as downward, rear-right, and rear-right are sequentially output. Due to the change in trajectory curvature of this stroke, the plane direction angle between adjacent key points spans multiple direction intervals. After direction stability determination, the system updates the broadcast content in time when the direction interval switches, outputs a pen-lift prompt after the end key point is reached, determines that all key points have been reached, and outputs a writing completion signal to terminate the current round of interactive guidance loop. The sound path of the above three strokes fully demonstrates the step-by-step key point progressive guidance process of the system from the starting key point of the first stroke to the ending key point of the last stroke, and verifies the collaborative work of each link in step 5 in the multi-stroke composite scenario.

[0057] Figure 8 is a schematic diagram of the system operation interface and writing result of the writing assistance teaching method for the blind according to the embodiments of the present application. As shown in Figure 8As shown, the system operation interface includes four functional areas: a text input area, a real-time video area, a visual output area, and a status display area. The text input area shows that the current content to be written is the Chinese character "大", the user starts the writing guidance process through the run button, triggers the calibration phase of the second step through the standard button, and terminates the current round of guidance through the stop button. The left side of the visual output area displays the target key point coordinate identifiers, showing the distribution pattern of the target key point sequence generated after the third step of skeleton thinning and ordered key point extraction for the content to be written in the virtual coordinate space. Each key point is presented in a stepped arrangement according to the stroke order and the sequential order within the stroke, which intuitively reflects the complete navigation path planning result after the system splits, resamples, and maps the three stroke segments of the Chinese character "大" to the virtual space. The real-time video area displays the real-time画面 of the writing area collected by the depth camera, in which the user's writing trace of the Chinese character "大" completed on the paper under the voice guidance of the system can be seen, and the trace is highly consistent in morphology with the character skeleton trajectory described by the target key point sequence. The bottom left of the status display area is marked with "Target Key Coordinate (18, 19)", which indicates the coordinate position of the currently activated key point in the target key point sequence in the virtual coordinate space. The middle part is marked with "Real-time Detection Coordinate (19, 18)", which indicates the virtual plane coordinate of the marker after the center coordinate of the marker in the current frame is mapped to the virtual space. The two values are highly close, indicating that the arrival distance of the current frame has met the preset arrival threshold condition. The right side is marked with "Writing Ended", which indicates that in the fifth step, the active pointer has crossed the boundary to determine that all key points have been reached and a writing completion signal has been output.

[0058] In summary, a writing-aided teaching method for the visually impaired according to the embodiments of the present application is clarified. It uses the virtual coordinate space as a unified intermediary to couple the real-time three-dimensional position perception of the writing tool with the ordered key point decomposition of the content to be written, establishes a unified correspondence between the real writing area and the target content through coordinate mapping, and further generates a direction movement prompt according to the plane relative relationship between the current position of the writing tool and the next target key point, and outputs a pen-up and pen-down prompt according to the deviation relationship between the real-time height of the writing tool and the reference height, so as to convert the invisible writing trajectory into an audible and followable step-by-step interactive guidance process, enabling blind and low-vision users to independently complete writing training without visual comparison and continuous assistance from others. This solution solves the problems that users do not know where to start writing and cannot stably advance along the target trajectory by splitting the content to be written into a sequence of key points with stroke order and guiding them point by point; through the height difference judgment mechanism in the vertical direction, the problem that users have difficulty perceiving whether the pen tip is in contact with the writing surface is solved; by introducing inter-frame timing constraints and adaptive fusion strategies in the depth calculation link, the interference of transient noise of the depth sensor on the pen-up and pen-down judgment is suppressed, and the timing consistency and physical rationality of the auditory guidance signal are guaranteed.

[0059] Further, a writing-aided teaching system for the visually impaired is also provided.

[0060] Figure 9 This is a block diagram of a blind writing-assisted teaching system according to an embodiment of this application. Figure 9 As shown, a blind writing assistance teaching system 100 according to an embodiment of this application includes: a marker detection and depth acquisition module 110, used to detect writing pen markers and acquire three-dimensional coordinates from collected writing area data to obtain marker center coordinates and marker depth values; a coordinate mapping and height calibration module 120, used to perform normalized coordinate mapping and contact height calibration on marker center coordinates and marker depth values ​​based on writing area boundary points to obtain virtual space mapping parameters, marker calibration height, and height fluctuation threshold; a key point extraction and spatial mapping module 130, used to extract and resample key points of the content to be written, and map them to a virtual coordinate space using virtual space mapping parameters to obtain a target key point sequence; and a real-time writing deviation solution module. The calculation module 140 is used to map the center coordinates of the marker frame by frame using virtual space mapping parameters to obtain the mapping result. The mapping result is then subtracted from the current key point in the target key point sequence, and the orientation angle and distance are calculated. Simultaneously, the height difference is quantized based on the marker depth value and the marker calibration height to obtain the planar orientation angle, arrival distance, and real-time height difference. The interactive prompt generation and iteration control module 150 is used to quantize the planar orientation angle and determine throttling stability based on the orientation interval division to generate planar movement prompts. The pen-raising and lowering status is determined based on the arrival distance and height fluctuation threshold to generate pen-raising and lowering prompts. When the arrival distance meets the arrival threshold, the current key point is switched to the subsequent key point in the target key point sequence until all key points have been reached.

[0061] In particular, in the embodiments of this application, a Braille writing-assisted teaching device is also provided, which is used to perform the above-described Braille writing-assisted teaching method.

[0062] Those skilled in the art will understand that the steps, measures, and schemes in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted; furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted; furthermore, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0063] The above-described embodiments are merely illustrative of several implementation methods of this disclosure, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent for the embodiments of this disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this disclosure, and these all fall within the protection scope of the embodiments of this disclosure. Therefore, the protection scope of the embodiments of this disclosure should be determined by the appended claims. As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and detail can be made without departing from the spirit and scope of the present invention as defined in the appended claims.

[0064] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for teaching Braille writing assistance, characterized in that, include: Step 1: Detect writing pen markers and obtain 3D coordinates from the collected writing area data to get the marker center coordinates and marker depth values; Step 2: Based on the boundary points of the writing area, normalize the coordinate mapping of the marker center and the marker depth value and perform contact height calibration to obtain virtual space mapping parameters, marker calibration height and height fluctuation threshold; Step 3: Extract and resample key points of the content to be written in an orderly manner, and map them to the virtual coordinate space using virtual space mapping parameters to obtain the target key point sequence; Step 4: Use virtual space mapping parameters to map the center coordinates of the marker frame by frame to obtain the mapping result. Subtract the mapping result from the current key point in the target key point sequence and calculate the orientation angle and distance. At the same time, quantize the height difference based on the marker depth value and the marker calibration height to obtain the planar orientation angle, arrival distance and real-time height difference. Step 5: Based on the division of the direction interval, the plane direction angle is quantified and throttling stability is determined to generate plane movement prompts. The real-time height difference is judged according to the arrival distance and height fluctuation threshold to generate pen lifting and lowering prompts. When the arrival distance meets the arrival threshold, the current key point is switched to the subsequent key point in the target key point sequence until all key points are reached.

2. The Braille writing-assisted teaching method according to claim 1, characterized in that, The data collected from the writing area includes color image frames of detectable markers on the writing pen and alignment depth image frames.

3. The Braille writing-assisted teaching method according to claim 2, characterized in that, Step one includes: Step 1.1: Input the color image frame into the pre-trained visual object detection model to perform bounding box regression and region recognition on the detectable markers on the writing pen to obtain the marker bounding boxes; Step 1.2: Based on the coordinates of the diagonal vertices of the bounding box of the marker, perform geometric center calculation on the bounding box region to obtain the coordinates of the marker center; Step 1.3: Using the center coordinates of the marker as the center, a local window of a preset pixel size is cropped on the aligned depth image frame. Zero-value invalid values ​​and statistical outliers are removed from the depth measurement values ​​of each pixel in the window to obtain an effective depth set. Step 1.4: Based on the coordinates of each pixel and the center of the marker, perform depth-weighted fusion on the effective depth set to obtain the marker depth value.

4. The Braille writing-assisted teaching method according to claim 1, characterized in that, Step two includes: Step 2.1: In the calibration phase, in response to the operation of placing the writing pen sequentially at the extreme boundary position of the writing area, the center coordinates of the markers in multiple consecutive frames are sampled and cached to construct a set of planar boundary coordinates. At the same time, the depth values ​​of the markers in the corresponding frames are height-mapped and cached to construct a set of contact height samples. Step 2.2: Based on the preset virtual resolution parameters, construct the virtual space mapping parameters by performing an affine transformation matrix on the steady-state coordinates of the upper left and lower right corners of the plane boundary coordinate set. Step 2.3: Perform baseline height estimation and tolerance adaptation on all sample values ​​in the contact height sample set to obtain the height fluctuation threshold.

5. The Braille writing-assisted teaching method according to claim 1, characterized in that, Step three includes: Step 3.1: Convert the content to be written into a reference character image using a font rendering engine, and then refine the skeleton of the reference character image to obtain a character skeleton image. Step 3.2: Perform basic sequence resampling on the character skeleton image to obtain the original pixel key point sequence; Step 3.3: Based on the affine transformation matrix in the virtual space mapping parameters, perform target coordinate system mapping calculation on each key point in the original pixel key point sequence to obtain the target key point sequence.

6. The Braille writing-assisted teaching method according to claim 1, characterized in that, Step four includes: Step 4.1: Using the affine transformation matrix in the virtual space mapping parameters, perform real-time virtual plane mapping on the center coordinates of the marker in the current frame to obtain the virtual plane coordinates of the marker; Step 4.2: Extract the coordinates of the currently active key point from the target key point sequence, and perform planar relative situation calculation on the virtual plane coordinates of the marker and the coordinates of the currently active key point to obtain the planar orientation angle and the distance to reach. Step 4.3: Calculate the algebraic difference between the current frame marker depth value and the marker calibration height to obtain the real-time height difference.

7. The Braille writing-assisted teaching method according to claim 1, characterized in that, Step five includes: Step 5.1: Based on the preset eight equally divided sector intervals, perform interval quantization index mapping on the planar direction angles, and generate planar movement prompts after time window throttling logic and continuous multi-frame stability determination. Step 5.2: When the distance reaches the preset arrival threshold range, the real-time height difference is compared with the height fluctuation threshold to determine the pen tip contact state and generate a pen lifting and lowering prompt. Step 5.3: Based on the comparison between the arrival distance and the preset arrival threshold, the active pointer in the target key point sequence is incrementally updated to switch to the subsequent key point. When the active pointer goes out of bounds, it is determined that all key points have been reached and a writing completion signal is output, terminating the current round of interactive guidance loop.

8. The Braille writing-assisted teaching method according to claim 3, characterized in that, Step 1.4 includes: Step 1.4.1: Based on the coordinates of each pixel and the center of the marker, perform intra-frame spatial weighting on the effective depth set to obtain intra-frame weighted depth values, and perform exponential smoothing prediction on the depth temporal sliding window to obtain temporal predicted depth values. Step 1.4.2: Quantitatively evaluate the temporal deviation factor of the intra-frame weighted depth value and the temporal predicted depth value to obtain the temporal deviation factor; Step 1.4.3: Based on the temporal deviation factor, the intra-frame weighted depth value and the temporal predicted depth value are adaptively linearly fused to obtain the marker depth value, and the marker depth value is pushed into the depth temporal sliding window to complete the iterative update.

9. A Braille writing-assisted teaching system, characterized in that, include: The marker detection and depth acquisition module is used to detect writing pen markers and acquire three-dimensional coordinates from the collected writing area data to obtain the marker center coordinates and marker depth values; The coordinate mapping and height calibration module is used to perform normalized coordinate mapping and contact height calibration on the marker center coordinates and marker depth values ​​based on the writing area boundary points to obtain virtual space mapping parameters, marker calibration height, and height fluctuation threshold. The key point extraction and spatial mapping module is used to extract and resample key points of the content to be written in an orderly manner, and to map them to a virtual coordinate space using virtual spatial mapping parameters to obtain the target key point sequence. The real-time writing deviation calculation module is used to map the center coordinates of the marker frame by frame using virtual space mapping parameters to obtain the mapping result. The mapping result is subtracted from the current key point in the target key point sequence and the orientation angle and distance are calculated. At the same time, the height difference is quantized according to the marker depth value and the marker calibration height to obtain the planar orientation angle, the distance to reach and the real-time height difference. The interactive prompt generation and iteration control module is used to quantify and throttle stability of the plane direction angle based on the direction interval division to generate plane movement prompts, determine the pen lifting and lowering status of the real-time height difference based on the arrival distance and height fluctuation threshold to generate pen lifting and lowering prompts, and switch the current key point to the subsequent key point in the target key point sequence when the arrival distance meets the arrival threshold until all key points have been reached.

10. A writing aid teaching device for the blind, characterized in that, include: The Braille writing aid teaching device is used to perform the Braille writing aid teaching method as described in any one of claims 1-8.

Images