Real-time Braille dot detection method and system for portable smart devices for visually impaired individuals

Abstract

This invention discloses a method and system for real-time Braille dot detection in portable smart devices for visually impaired individuals, belonging to the field of Braille dot detection technology. The system executes the steps of the method, specifically employing a lightweight deep convolutional neural network model for rapid Braille dot detection in images. This model is based on the MobileNetV3 model, incorporating a hybrid depthwise separable convolutional module and a channel attention mechanism. It combines an adaptive image preprocessing module to automatically adjust image brightness and contrast, and multi-scale Braille dot feature extraction technology to fuse features of different sizes, thereby achieving accurate and stable recognition of Braille dots in various environments. The system also provides real-time audio feedback to the user, meeting the real-time and accurate recognition needs of visually impaired individuals in complex scenarios.

Description

Technical Field

[0001] This invention belongs to the field of Braille dot detection technology, specifically, it relates to a method and system for real-time Braille dot detection in portable smart devices for visually impaired individuals. Background Technology

[0002] Existing Braille dot detection technologies primarily rely on traditional image processing methods and simple machine learning algorithms. However, these technologies suffer from numerous shortcomings in practical applications. They are generally sensitive to environmental factors such as lighting and angle, leading to unstable recognition results. In real-world scenarios, lighting intensity can vary drastically depending on the shooting environment; for example, low indoor light or strong outdoor light conditions can severely impact image quality, making it difficult for edge detection and thresholding to accurately extract Braille dot features. Similarly, deviations in shooting angle can cause distortion of Braille dot shapes, reducing the recognition accuracy of machine learning algorithms. Secondly, existing technologies struggle to effectively handle Braille dot features of different sizes and shapes. Braille dots on different document carriers vary in size, and traditional methods, due to limitations in feature extraction, cannot adapt to this diversity, thus failing to comprehensively and accurately identify various Braille dots.

[0003] Furthermore, the recognition speed of existing technologies is insufficient to meet the requirements of real-time feedback. Traditional methods involve significant computational demands when processing complex images, especially when the image resolution is high or a large number of consecutive frames need to be processed, resulting in a substantial decrease in recognition speed and an inability to provide immediate feedback for visually impaired individuals. For example, when processing high-resolution Braille images, traditional methods may require several seconds or even tens of seconds to complete recognition, which is far from adequate for real-time assisted reading. Summary of the Invention

[0004] The purpose of this invention is to propose a method and system for real-time Braille dot detection in portable smart devices for visually impaired individuals. This method is suitable for scenarios where visually impaired individuals use handheld smart devices to read Braille text. The invention utilizes a lightweight deep convolutional neural network model for rapid Braille dot detection in images, combined with an adaptive image preprocessing module to automatically adjust image brightness and contrast, and multi-scale Braille dot feature extraction technology to fuse features of different sizes. This achieves accurate and stable recognition of Braille dots in various environments, and the recognition results are broadcast to the user in real time via a voice feedback module, meeting the real-time and accurate recognition needs of visually impaired individuals in complex scenarios.

[0005] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0006] According to a first aspect of the present invention, a method for real-time Braille dot detection in portable smart devices for visually impaired individuals is proposed, comprising:

[0007] Step 1: Adaptive image preprocessing: Obtain ambient light intensity, dynamically adjust image brightness and contrast, and combine multi-scale filter fusion and image normalization processing;

[0008] Step 2: Construct and train a lightweight deep convolutional neural network model: The lightweight deep convolutional neural network model uses an improved MobileNetV3 model for Braille dot detection. The improved MobileNetV3 model is based on the original MobileNetV3 model and introduces a hybrid depthwise separable convolutional module and a channel attention mechanism.

[0009] Step 3: Multi-scale feature fusion: Extract multi-scale feature maps from different layers of the improved MobileNetV3 model, and achieve feature fusion by adding them element by element;

[0010] Step 4: Real-time target detection: An anchor-box-based detection algorithm is applied to the fused feature map. The Braille point locations are filtered by Intersection over Union (IoU), and Non-maximum Suppression (NMS) is used to remove redundant boxes.

[0011] Step 5: Voice Feedback

[0012] The test results are converted into voice signals and broadcast.

[0013] Step 6: System Iterative Optimization

[0014] Optimize model parameters based on user feedback and usage data.

[0015] Furthermore, step 1 includes:

[0016] Step 101: Ambient light perception and adaptive brightness adjustment: The ambient light intensity value is obtained through a light sensor, and the brightness and contrast of the image are dynamically adjusted according to the ambient light intensity value;

[0017] Step 102: Multi-scale filter fusion: Image enhancement is performed using Gaussian filter, Laplacian filter and adaptive histogram equalization;

[0018] Step 103: Image normalization and standardization: Normalize and standardize the preprocessed image.

[0019] Furthermore, in step 101, the brightness and contrast adjustment formulas are selected based on the ambient light intensity value:

[0020] Under low light conditions, i.e. At that time, the formula for increasing brightness is:

[0021] ;

[0022] in, For the position of the image after brightness enhancement Pixel values; For the original image at position Pixel values; This is the brightness enhancement factor; This is the critical value under low light conditions; This represents the current ambient light intensity value.

[0023] Under low light conditions, i.e. When the contrast enhancement formula is:

[0024] ;

[0025] in, For the image after contrast adjustment, in position Pixel values; For the position of the image after brightness enhancement Pixel values; This is the contrast enhancement factor;

[0026] Under strong light conditions, i.e. When the brightness decreases, the formula is:

[0027] ;

[0028] in, For the image after brightness reduction at position Pixel values; For the original image at position Pixel values; This is the brightness reduction factor; This represents the current ambient light intensity value. This is the critical value under strong light conditions;

[0029] Under strong light conditions, i.e. At that time, the contrast adjustment formula is:

[0030] ;

[0031] in, For the image after contrast adjustment, in position Pixel values; For the image after brightness reduction at position Pixel values;

[0032] Under normal lighting conditions, i.e. At this time, only fine-tuning of contrast is performed:

[0033] ;

[0034] in, For the image after contrast adjustment, in position Pixel values; For the original image at position The pixel value.

[0035] Furthermore, in the improved MobileNetV3 model, the hybrid depthwise separable convolution module first performs a depthwise convolution operation on the input feature map, and then divides the depthwise convolutioned feature map into... Each group undergoes independent pointwise convolution, concatenating the feature maps of all groups. An additional pointwise convolution is then used to fuse the concatenated feature maps, resulting in a fused output feature map. A lightweight channel attention mechanism (CAM) is added after each group's convolution.

[0036] Further, model training: collect and label Braille text images, and train them using a cross-entropy loss function with class-balanced weights; the formula for the cross-entropy loss function with class-balanced weights is:

[0037] ;

[0038] in To balance the value of the loss function; For the first Class-balanced weights for each sample; For the first The actual label of each sample, where a label of 1 represents a positive sample and a label of 0 represents a negative sample; For the model to predict the first The probability that a sample belongs to a positive sample; For the first The negative sample weights of each sample are used to adjust the loss contribution of the negative samples.

[0039] The model parameters were adjusted using the Adam optimization algorithm, with an initial learning rate of 0.001 and a batch size of 32.

[0040] Furthermore, step 3 includes:

[0041] Step 301: Feature extraction, extracting multi-scale feature maps at different levels of the improved MobileNetV3 model;

[0042] Step 302: Feature fusion, which uses an element-wise addition method to fuse feature maps of different scales.

[0043] Furthermore, step 4 includes:

[0044] Preset anchor frames of different sizes to match Braille dot sizes;

[0045] Calculate the intersection-union ratio (IoU) between the anchor box and the ground truth bounding box, and filter candidate boxes with IoU ≥ 0.5;

[0046] Non-maximum suppression (NMS) is used to preserve the highest confidence detection boxes.

[0047] Furthermore, step 5 includes:

[0048] Step 501: Recognition Result Conversion: Convert the detected Braille dot information into speech content;

[0049] Step 502: Speech Synthesis and Broadcasting: Use a speech synthesis engine to synthesize the speech text into a speech signal and broadcast it.

[0050] According to a second aspect of the present invention, a real-time Braille dot detection system for portable smart devices for visually impaired individuals is provided. This system is used to perform the steps of the real-time Braille dot detection method for portable smart devices for visually impaired individuals, including:

[0051] Adaptive image preprocessing module: Equipped with a light sensor and an image processing unit, it is used to automatically adjust the image brightness and contrast according to changes in ambient light.

[0052] Lightweight deep convolutional neural network module: Deploys an improved MobileNetV3 model for Braille dot detection on preprocessed images; the improved MobileNetV3 model is based on the original MobileNetV3 model, introducing a hybrid depthwise separable convolutional module and a channel attention mechanism.

[0053] Multi-scale Braille dot feature extraction module: Constructs a feature pyramid structure to fuse features from different levels;

[0054] The voice feedback module is used to immediately convert the recognition result into speech and broadcast it to the user after detecting Braille dots.

[0055] The beneficial effects of this invention are as follows: This invention utilizes the collaborative work of an adaptive image preprocessing module, a lightweight deep convolutional neural network module, a multi-scale Braille dot feature extraction module, and a voice feedback module. Adaptive image preprocessing and a lightweight deep convolutional neural network model are used to quickly detect Braille dots in the processed image. This model is based on the MobileNetV3 model, incorporating a hybrid depthwise separable convolutional module and a channel attention mechanism. Combined with the adaptive image preprocessing module automatically adjusting image brightness and contrast, and the multi-scale Braille dot feature extraction technology fusing features of different sizes, this invention achieves real-time Braille dot detection and voice feedback under complex environments such as varying lighting and angles. Compared to existing technologies, this invention can stably recognize Braille dots in various practical scenarios, unaffected by significant environmental factors such as lighting and angle. It can effectively handle Braille dot features of different sizes and shapes, improving the accuracy and comprehensiveness of recognition. Furthermore, this invention has a fast recognition speed and can achieve real-time feedback, helping visually impaired individuals quickly obtain Braille information. This not only enhances the reading experience for visually impaired individuals, enabling them to read Braille texts more conveniently and efficiently, but also provides important support for them in education, work, and daily life. Attached Figure Description

[0056] Figure 1 Flowchart of a method for real-time Braille dot detection in portable smart devices for visually impaired individuals;

[0057] Figure 2 A schematic diagram illustrating the working principle of a portable smart device real-time Braille dot detection system for visually impaired individuals;

[0058] Figure 3 A flowchart illustrating the construction and training process of a lightweight deep convolutional neural network model;

[0059] Figure 4 Flowchart for multi-scale Braille dot feature extraction;

[0060] Figure 5 Flowchart for real-time detection of Braille dots. Detailed Implementation

[0061] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, this invention is not limited to the following embodiments, and specific implementation methods can be determined according to the technical solutions of this invention and actual circumstances. To avoid obscuring the essence of this invention, well-known methods, processes, and procedures are not described in detail.

[0062] like Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 As shown, this invention proposes a real-time Braille dot detection system for portable smart devices for visually impaired individuals. This system executes the steps of a real-time Braille dot detection method for portable smart devices for visually impaired individuals, specifically including:

[0063] An adaptive image preprocessing module is used to automatically adjust the image brightness and contrast according to changes in ambient light, providing stable image input for subsequent Braille dot detection;

[0064] A lightweight deep convolutional neural network module is used to detect Braille dots in preprocessed images. Its lightweight structure ensures efficient operation of the model on handheld smart devices.

[0065] The multi-scale Braille dot feature extraction module is used to extract Braille dot features of different sizes and fuse these features to improve the model's ability to recognize Braille dots of various shapes and sizes.

[0066] The voice feedback module is used to immediately convert the recognition result into speech and broadcast it to the user after detecting Braille dots, helping visually impaired people to quickly obtain Braille information.

[0067] The adaptive image preprocessing module includes:

[0068] A light sensor is used to detect ambient light intensity in real time.

[0069] The image processing unit is used to dynamically adjust the brightness and contrast of the image based on the data detected by the light sensor.

[0070] The lightweight deep convolutional neural network module uses MobileNetV3 as the basic network structure and introduces hybrid depthwise separable convolutional modules and channel attention mechanisms to reduce computational complexity.

[0071] The multi-scale Braille dot feature extraction module is implemented through a multi-level feature pyramid structure to simultaneously capture Braille dot features at both small and large scales.

[0072] The present invention proposes a method for real-time Braille dot detection in portable smart devices for visually impaired individuals, comprising:

[0073] Step 1: Adaptive Image Preprocessing

[0074] Step 101: Ambient Light Sensing and Adaptive Brightness Adjustment

[0075] To adapt to different lighting conditions, this invention designs an adaptive brightness adjustment algorithm. First, the ambient light intensity value is obtained through a light sensor. (Range 0-255, dimensionless, is the relative intensity value output by the light sensor). According to The value dynamically adjusts the brightness and contrast of the image.

[0076] Low light conditions When ), the brightness enhancement formula is:

[0077] ;

[0078] in, For the position of the image after brightness enhancement Pixel values; For the original image at position Pixel values; The brightness enhancement factor represents the intensity of the brightness increase; the larger the value, the more significant the brightness increase. In this embodiment... The value is 0.3; This is the critical value for low-light conditions, in this embodiment... The value is 50; This represents the current ambient light intensity value, typically ranging from 0 to 255. A lower value indicates a darker environment. Under low-light conditions (i.e., ambient light intensity value...), this value represents the current ambient light intensity. ), by calculating the current ambient light intensity value With critical value The difference between the two values ​​is multiplied by a brightness enhancement factor of 0.3 to obtain the brightness value that needs to be increased. Finally, this brightness value is added to the pixel value of the original image to achieve the image brightness enhancement. It should be noted that since both are dimensionless values, representing relative brightness or intensity, they can be directly added together.

[0079] The formula for enhancing contrast is:

[0080] ;

[0081] in, For the image after contrast adjustment, in position Pixel values; For the position of the image after brightness enhancement Pixel values; The contrast enhancement factor represents the strength of the contrast enhancement. A value greater than 1 indicates enhanced contrast; the larger the value, the more significant the contrast enhancement. In this embodiment... The value is 1.2. Based on the increased brightness, each pixel value is multiplied by a contrast enhancement factor of 1.2, thereby widening the difference between pixel values, enhancing image contrast, and making image details clearer.

[0082] Strong light conditions ( When the brightness decreases, the formula is:

[0083] ;

[0084] in For the image after brightness reduction at position Pixel values; For the original image at position Pixel values; This is the brightness reduction coefficient, which indicates the intensity of the brightness reduction. The larger the value, the more significant the brightness reduction. The value is 0.2; This is the current ambient light intensity value, typically ranging from 0 to 255. It is a dimensionless value representing relative light intensity; the larger the value, the brighter the environment. This is the critical value for strong light conditions, in this embodiment The value is 200;

[0085] The formula for adjusting contrast is:

[0086] ;

[0087] in, For the image after contrast adjustment, in position Pixel values; For the image after brightness reduction at position The pixel value.

[0088] Under normal lighting conditions, i.e. At this time, only fine-tuning of contrast is performed:

[0089] ;

[0090] in, For the image after contrast adjustment, in position Pixel values; For the original image at position The pixel value.

[0091] Step 102: Multi-scale filter fusion

[0092] To further improve image quality, this invention designs a multi-scale filter fusion algorithm that combines a Gaussian filter, a Laplace filter, and an adaptive histogram equalization (CLAHE) algorithm.

[0093] Gaussian filtering: The image is smoothed using a Gaussian filter with a kernel size of 5×5 and a standard deviation of 1.5 for the Gaussian filter kernel.

[0094] ;

[0095] in, The image after Gaussian filtering smoothing is located at... Pixel values; Indicates Gaussian filtering. This represents the standard deviation, which is taken as 1.5 here. For the image after contrast adjustment, in position Pixel values;

[0096] Laplacian filtering: Applying a Laplacian filter to an image smoothed by Gaussian filtering enhances edge information.

[0097] ;

[0098] in For the image after Laplacian filtering, at position Pixel values; The image after Gaussian filtering smoothing is located at... Pixel values; " indicates a convolution operation; matrix This represents the Laplacian filter kernel, used to detect edge information in an image;

[0099] Weighted fusion:

[0100] ;

[0101] in, For the merged image at position Pixel values; The image after Gaussian filtering smoothing is located at... Pixel values; For the image after Laplacian filtering, at position Pixel values; The weighting coefficients represent the weights of the Laplacian filtering results in the fused image. In this embodiment, The value is 0.7.

[0102] Adaptive Histogram Equalization (CLAHE): The CLAHE algorithm is applied to the fused image to enhance local contrast; the contrast limit of CLAHE is set to 2.0, and the block size is 8×8.

[0103] ;

[0104] in, For the image after adaptive histogram equalization at the location Pixel values; For the merged image at position The pixel value; CLAHE represents the adaptive histogram equalization algorithm, which is an existing technology.

[0105] The contrast limit is set to 2.0 to limit the enhancement of local contrast and prevent excessive amplification of noise.

[0106] Setting the block size to 8×8 means dividing the image into 8×8 blocks for local histogram equalization.

[0107] Step 103: Image Normalization and Standardization

[0108] Normalization and standardization are performed on the preprocessed images to improve the training efficiency and stability of subsequent deep learning models.

[0109] Normalization: Adjusts the pixel value range to [0, 1]:

[0110] ;

[0111] in, For the normalized image at position The pixel value ranges from [0,1]. For the image after adaptive histogram equalization, at position The pixel values ​​are originally in the range [0, 255].

[0112] Standardization: Adjust the mean of the image to 0 and the standard deviation to 1;

[0113] ;

[0114] in, For the standardized image at position The pixel values ​​typically have a mean of 0 and a standard deviation of 1. For the normalized image at position The pixel value ranges from [0,1].

[0115] This is the average value (mean) of all pixel values ​​in the normalized image. This is the standard deviation of all pixel values ​​in the normalized image.

[0116] Step 2: Construction and Training of a Lightweight Deep Convolutional Neural Network Model

[0117] Step 201: Model Building

[0118] To achieve efficient Braille dot detection, this invention deeply modifies MobileNetV3, introducing a novel hybrid depthwise separable convolutional structure and attention mechanism module.

[0119] Hybrid Depthwise Separable Convolution (HDSC) Module: Based on MobileNetV3, a hybrid depthwise separable convolution module was designed. The hybrid depthwise separable convolution module combines the advantages of standard convolution and depthwise separable convolution, and introduces group convolution to further reduce the amount of computation.

[0120] ;

[0121] in, The input feature map is typically a three-dimensional tensor with a size of 1. , The height of the feature map. Width is the width of the feature map. The number of channels in the feature map. The output feature map is the result of depthwise convolution, and its size is the same as the input. ;

[0122] DepthwiseConv is a depthwise convolution operation, meaning that a convolution operation is performed on each input channel separately, with each channel using an independent convolution kernel. The kernel size is typically [size missing]. ,in, The spatial size of the convolution kernel, such as 3 or 5.

[0123] Grouped convolution: Divides the feature map after depthwise convolution into groups. The input feature map is divided into groups, and each group undergoes pointwise convolution operations independently. Let the number of channels in the input feature map be... Number of groups Then the number of channels in each group is :

[0124] ;

[0125] in, The first feature map after depthwise convolution is the... A subset of the group's feature maps; For the first The output feature map of the group after pointwise convolution;

[0126] This is pointwise convolution, which uses a 1×1 convolution kernel to perform the convolution operation and is used to fuse channel information; The number of groups is given in the image. ; Given the number of channels in the input feature map, the number of channels per group is... .

[0127] Feature fusion: The feature maps of all groups are concatenated and then fused using an additional pointwise convolution.

[0128] ;

[0129] in, For the first The output feature map of the group after pointwise convolution;

[0130] This means concatenating the feature maps of all groups along the channel dimension; This is pointwise convolution, which uses a 1×1 convolution kernel to perform convolution operations and fuse the concatenated feature maps. This is the final fused output feature map.

[0131] Attention Mechanism Module: A lightweight Channel Attention Mechanism (CAM) is introduced after the HDSC module to enhance the perception of key features.

[0132] Global average pooling: for the input feature map Perform global average pooling to obtain the feature vector for each channel:

[0133] ;

[0134] in, The input feature map has a size of [size missing]. , The height of the feature map. Width is the width of the feature map. The number of channels in the feature map. This is a global average pooling operation, which performs global average pooling on the feature map of each channel, that is, it averages all pixel values ​​of each channel. The feature vector after global average pooling has a length of Each element corresponds to the average value of a channel; It is the input feature map The feature values ​​of the first channel after global average pooling; It is the input feature map The feature value of the second channel after global average pooling; It is the input feature map The feature value of the third channel after global average pooling.

[0135] Fully connected layer and activation function: Channel features are encoded using a shared fully connected layer (FC) and ReLU activation function.

[0136] ;

[0137] in The feature vector after global average pooling has a length of ; For operations on a fully connected layer, the length is... The input feature vector is mapped to a feature vector of another length (usually the same or shorter). For activation functions; For passing through the fully connected layer and Feature vectors after activation function;

[0138] Sigmoid activation function: generates channel attention weights :

[0139] ;

[0140] in Attention weights are used to emphasize important features; Activation functions are used to convert values ​​into weights between 0 and 1; This is a fully connected layer used to process features.

[0141] Feature enhancement: The attention weights are multiplied channel-by-channel by the original feature map to obtain the enhanced feature map.

[0142] ;

[0143] Where X This is the original feature map, which is the input data for neural network processing; These are attention weights, representing the importance of each feature; This is an element-wise multiplication operation used to apply weights to the feature map; The enhanced feature map is obtained by emphasizing important features.

[0144] Step 202: Model Training

[0145] Data collection and annotation: Collect 10,000 Braille text images under different lighting and angle conditions, and annotate them with the bounding box coordinates of the Braille dots. And categories (such as Braille characters like "A", "B" etc.). The x-coordinate of the top-left corner of the bounding box. The ordinate of the top-left corner of the bounding box. The x-coordinate of the bottom right corner of the bounding box. This is the ordinate of the bottom right corner of the bounding box.

[0146] Loss function design: An improved cross-entropy loss function is adopted, combined with class balancing weights, to address the class imbalance problem.

[0147] ;

[0148] in To balance the value of the loss function, used to handle class imbalance problems. For the first The class balance weights for each sample are used to adjust the influence of different class samples in the loss calculation, in order to solve the class imbalance problem. For the first The actual label (0 or 1) of each sample represents the true category; For the model to predict the first The probability that a sample belongs to a positive sample (label 1); For the first The negative sample weights of each sample are used to adjust the loss contribution of the negative samples.

[0149] The model parameters were tuned using the Adam optimization algorithm, with an initial learning rate of 0.001 and a batch size of 32. During training, data augmentation techniques (such as random rotation and scaling) were used to expand the dataset and improve the model's generalization ability.

[0150] With this modified lightweight deep convolutional neural network module, the model can run efficiently on handheld smart devices, while significantly improving the accuracy and stability of Braille dot detection, providing more reliable input for subsequent multi-scale feature extraction and speech feedback modules.

[0151] Step 3: Multi-scale Braille dot feature extraction

[0152] Step 301: Feature Extraction

[0153] Multi-scale feature maps are extracted at different levels of the MobileNetV3 model. For example, low-level features such as edges and textures are extracted in the shallow layers of the model, while more semantic Braille dot features are extracted in the deeper layers. By applying a Feature Pyramid Network (FPN) structure to the feature maps at different levels, features at different scales are fused to generate a fused feature map. This feature map contains rich information about Braille dots of different sizes, which helps improve the model's ability to recognize Braille dots of various shapes and sizes. Let the feature maps extracted from the shallow to the deep layers be as follows: (Size is 64×64×32) (Size is 32×32×64) (Size is 16×16×128), its size gradually decreases, and the number of channels gradually increases; firstly, for Perform upsampling to make its size consistent with... The same (upsampling method is bilinear interpolation), then and after upsampling By adding elements one by one, a fused feature map is obtained. (Size is 64×64×96); Similarly, will After upsampling and Adding each element one by one yields the final fused feature map. (Size is 64×64×160); This fusion strategy makes full use of the information from different levels of feature maps, enhancing the model's ability to express multi-scale Braille dot features.

[0154] Step 302: Feature Fusion

[0155] Feature maps of different scales are fused using an element-wise addition method. For example, for and after upsampling The specific formula for adding elements one by one is:

[0156] ;

[0157] Where x and y represent the spatial locations of the feature map. This represents the feature map after upsampling. It is a feature map In position The value, yes and after upsampling The fused feature map obtained by element-wise addition is located at... The value of . In this way, feature information at different scales is fused together, so that the fused feature map can contain both low-level and high-level features, thereby improving the model's ability to detect Braille dots.

[0158] Step 4: Real-time detection of Braille dots

[0159] Step 401: Braille dot detection

[0160] An anchor-based object detection algorithm is applied to the fused feature map. A series of anchor boxes with different scales and aspect ratios are generated and matched with the ground truth bounding boxes of the Braille dots. For example, the scale range of the anchor boxes is set to [16, 64], and the aspect ratio is [0.5, 1, 2], generating multiple anchor boxes. The anchor boxes that match the Braille dots are selected by calculating the intersection-over-union (IoU) ratio between the anchor boxes and the ground truth bounding boxes. The formula for calculating IoU is:

[0161] ;

[0162] in This represents the area of ​​the overlapping region between the anchor frame and the actual bounding box. This represents the area of ​​the union of the anchor box and the ground truth bounding box. For example, given an anchor box and a ground truth bounding box, , ,but The IoU threshold is set to 0.5. When the IoU between the anchor box and the real bounding box is greater than or equal to 0.5, the anchor box is considered to match the real bounding box.

[0163] Step 402: Post-processing operations

[0164] The detection results are processed using Non-Maximum Suppression (NMS) to remove redundant detection boxes and retain the bounding boxes most likely to contain Braille dots. An NMS threshold of 0.5 is set; that is, when the IoU between two detection boxes is greater than 0.5, the detection box with the higher score is retained, and other detection boxes are suppressed. This ultimately yields the precise location and category information of the Braille dots. For example, after NMS processing, 10 bounding boxes most likely to contain Braille dots are selected from 100 initial detection boxes. Each bounding box contains the category probability and coordinate information of the Braille dot.

[0165] Step 5: Voice feedback module triggered

[0166] Step 501: Recognition Result Conversion: Convert the detected Braille dot information into speech content; for example, convert the characters or words corresponding to the Braille dots into speech text. If the detected Braille dot characters are "A", "B", and "C", then the speech text is "ABC".

[0167] Step 502: Speech Synthesis and Broadcasting: Using the speech synthesis engine of the smart device (such as Google Text-to-Speech or other open-source speech synthesis libraries), the speech text is synthesized into a natural and fluent speech signal, which is then broadcast to the user in real time through the device's speaker. The speech rate and volume can be adjusted according to the user's preferences to ensure that the user can clearly hear the feedback information. For example, the user can set the speech rate to 150-200 words per minute and the volume to 70%-80% of the device's maximum volume.

[0168] Step 6: System Optimization and Evaluation

[0169] The system is iteratively optimized by continuously collecting user feedback and data from real-world usage scenarios. For example, based on user feedback regarding recognition errors, model parameters are adjusted, and image preprocessing algorithms and feature extraction strategies are optimized. For instance, user feedback indicated low accuracy in recognizing Braille dots under strong light. Analysis revealed that overexposure of the image caused the loss of Braille dot features. Therefore, the parameters of the adaptive image preprocessing module were adjusted to reduce the brightness of images under strong light and enhance contrast. After retraining the model, the recognition accuracy under strong light improved from 8%.

[0170] The system's performance is evaluated using metrics such as precision, recall, and F1 score. It's important to note that the F1 score is a comprehensive metric used in machine learning to evaluate the performance of classification models; it is the harmonic mean of precision and recall, and is particularly suitable for scenarios with imbalanced samples. Simultaneously, the system's response time is recorded to ensure its real-time performance meets user requirements.

[0171] In actual testing, the system detected 950 Braille dots out of 1000 Braille text images, with 900 being correct and 50 being incorrect. The actual images contained 920 Braille dots. Therefore, the accuracy is approximately 900 / 950 ≈ 94.7%, the recall is approximately 900 / 920 ≈ 97.8%, and the F1 score is approximately 2 × (94.7% × 97.8%) / (94.7% + 97.8%) ≈ 96.2%. The average response time was calculated through multiple tests and is required to be less than 1 second.

[0172] The table below shows the system performance evaluation results under different lighting conditions (example):

[0173]

[0174] Furthermore, as a further improvement of this invention, the system interface includes a camera button, a voice broadcast switch, and settings options. Users can operate the system via touch screen or voice commands. For example, they can long-press the camera button to take a picture of the Braille text, tap the voice broadcast switch to turn voice feedback on or off, and enter the settings options to adjust parameters such as speech rate and volume. Users can personalize the settings according to their preferences. For example, users can set the language of the voice broadcast (e.g., Chinese, English), speech rate (100-250 words per minute), and volume. The system saves the user's settings and applies them automatically in subsequent use. In addition, users can choose whether to enable vibration feedback; when Braille dots are detected, the device will vibrate to enhance the user's sensory experience.

[0175] In summary, this invention features rapid recognition speed and real-time feedback, helping visually impaired individuals quickly access Braille information. This not only enhances the reading experience for visually impaired individuals, enabling them to read Braille texts more conveniently and efficiently, but also provides crucial support in education, work, and daily life. Through a user interface and personalized settings, this invention also caters to the operating habits and needs of different users, improving system usability and user satisfaction, thus possessing significant social and practical value.

Claims

1. A portable smart device real-time Braille dot detection method for visually impaired people, characterized in that, Comprise: Step 1: adaptive image preprocessing: obtain ambient light intensity, dynamically adjust image brightness and contrast, combine multi-scale filter fusion and image standardization processing; Step 2: constructing and training a lightweight deep convolutional neural network model: the lightweight deep convolutional neural network model adopts an improved MobileNetV3 model for Braille point detection, the improved MobileNetV3 model is introduced into a mixed depth separable convolution module and a channel attention mechanism on the basis of the original MobileNetV3 model; in the improved MobileNetV3 model, the mixed depth separable convolution module first performs a depth convolution operation on the input feature map, then divides the feature map after the depth convolution into groups, each group independently performs a point-by-point convolution operation, splices the feature maps of all groups, and fuses the spliced feature maps through an additional point-by-point convolution to obtain a fused output feature map; a lightweight channel attention mechanism CAM is added after each group convolution; Step 3: multi-scale feature fusion: extract multi-scale feature maps from different levels of the improved MobileNetV3 model, and realize feature fusion by element-wise addition; Step 4: real-time target detection: apply anchor-based detection algorithm on the fused feature map, filter Braille point position through intersection over union IoU, and remove redundant boxes by non-maximum suppression NMS; Step 5: voice feedback: Convert the detection result into a voice signal and broadcast it; Step 6: system iterative optimization: Optimize model parameters based on user feedback and usage data.

2. The portable smart device real-time Braille dot detection method for visually impaired people according to claim 1, wherein, The step 1 comprises: Step 101: ambient light perception and adaptive brightness adjustment: obtain ambient light intensity value through light sensor, and dynamically adjust brightness and contrast of image according to ambient light intensity value; Step 102: multi-scale filter fusion: image enhancement is performed by using Gaussian filter, Laplace filter and adaptive histogram equalization; Step 103: image normalization and standardization: normalize and standardize the preprocessed image.

3. The portable smart device real-time Braille dot detection method for visually impaired people according to claim 2, characterized in that, In step 101, brightness and contrast adjustment formula is selected according to ambient light intensity value: In low light conditions, i.e. the brightness enhancement formula is: ； wherein, is the pixel value of the image after brightness enhancement at position ; is the pixel value of the original image at position ; is the brightness enhancement coefficient; is the critical value of the low light condition; is the current ambient light intensity value; In low light conditions, i.e. The contrast enhancement formula is: ； wherein, is the pixel value of the image after contrast adjustment at position ; is the pixel value of the image after brightness enhancement at position ; is a contrast enhancement coefficient. Under strong light conditions, i.e. the luminance reduction formula is: ； wherein, is a pixel value of the image after luminance reduction at position ; is a pixel value of the original image at position ; is a luminance reduction coefficient; is a current ambient light intensity value; is a critical value for strong light conditions; In strong light conditions, i.e. The contrast adjustment formula is: ； wherein, is the pixel value of the image after contrast adjustment at position ; is the pixel value of the image after brightness reduction at position ; Under normal lighting conditions, i.e. only a contrast fine tuning is performed: ； wherein is the pixel value of the contrast-adjusted image at position ; is the pixel value of the original image at position . 4.The real-time Braille dot detection method for visually impaired people using portable smart device of claim 1, wherein, Model training: collect and label Braille text images, and train using cross-entropy loss function with class balance weight; the formula of the cross-entropy loss function with class balance weight is: ； wherein is a value of a balanced loss function; is a class balance weight for the th sample; is an actual label for the th sample, where a label of 1 indicates a positive sample and a label of 0 indicates a negative sample; is a probability that the model predicts the th sample belongs to a positive sample; is a negative sample weight for the th sample, used to adjust the loss contribution of negative samples; Use Adam optimization algorithm to adjust model parameters, initial learning rate is 0.001, and batch size is 32. 5.The real-time Braille dot detection method for visually impaired people using portable smart device of claim 1, wherein, The step 3 comprises: Step 301: feature extraction, multi-scale feature maps are extracted at different levels of the improved MobileNetV3 model; Step 302: feature fusion, different scale feature maps are fused by element-wise addition. 6.The real-time Braille dot detection method for visually impaired people using portable smart device of claim 1, wherein, The step 4 comprises: Pre-set anchor boxes of different scales match the size of Braille points; Calculate the intersection over union IoU of anchor boxes and real bounding boxes, and filter the candidate boxes with IoU≥0.5; Use non-maximum suppression NMS to retain the highest confidence detection box.

7. The portable smart device real-time Braille dot detection method for visually impaired people according to claim 1, wherein, The step 5 comprises: Step 501: recognition result conversion: convert the detected Braille point information into voice content; Step 502: speech synthesis and broadcast: use speech synthesis engine to synthesize voice text into voice signal and broadcast.

8. A portable smart device real-time Braille dot detection system for visually impaired people, characterized in that, The system is used to execute the steps of the real-time Braille point detection method for visually impaired people's portable intelligent device in any one of claims 1-7, comprising: Adaptive image preprocessing module: configured with light sensor and image processing unit, used to automatically adjust image brightness and contrast according to ambient light change; Lightweight deep convolutional neural network module: deploy improved MobileNetV3 model for Braille point detection on preprocessed image; the improved MobileNetV3 model is based on the original MobileNetV3 model, which introduces mixed depth separable convolution module and channel attention mechanism; Multi-scale Braille point feature extraction module: construct feature pyramid structure to fuse features at different levels; Voice feedback module: for converting the recognition result into voice and playing it to the user immediately after detecting the Braille point.

Images