Double-sided braille image recognition method and system based on fusion language model and target detection

Abstract

The invention discloses a double-sided braille image recognition method and system based on a fusion language model and target detection, and the method comprises the steps: obtaining to-be-recognized braille image data which is single-sided braille image data containing front braille information and back braille information; inputting the single-side braille image data of any side into a double-side braille detection model for processing to obtain a double-side detection result; wherein the double-sided braille detection model is obtained by training a double-sided braille image recognition data set, and the double-sided detection result comprises position and category information of front braille convex points and back braille concave points. According to the method, the accuracy of the final recognition result is improved, adaptive adjustment can be carried out according to braille alphabets of different languages, and the flexibility of target detection is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a double-sided braille image recognition method and system based on a fusion language model and target detection. BACKGROUND

[0002] With the growing demand for braille resources from the large visually impaired population, braille publishing and usage have become increasingly widespread. However, the production and acquisition of braille resources still face many challenges, among which the quality control problem in the production process of braille embossed products is particularly prominent. Unlike traditional printed matter, braille embossed products must undergo strict manual proofreading after production to ensure accuracy. This proofreading process is extremely complex and requires professional personnel to identify braille content through tactile perception combined with visual recognition. Due to the small size of braille dots, a single proofreading cannot guarantee complete accuracy, and multiple rounds of proofreading are often required, which is time-consuming and labor-intensive, severely restricting the production efficiency of braille resources such as braille embossed products.

[0003] The difficulty in equipment maintenance further exacerbates the problem. When the braille embosser breaks down, the problem is more difficult: first, the braille embosser is expensive and the replacement cost is high; second, the braille embossing technology has not been widely popularized, making it difficult to obtain repair services; in addition, even if the repair personnel have the corresponding repair skills, the spare parts of the braille embosser lack stable supply channels, and it is difficult to find relevant manufacturers, and even if successful ordering, the delivery cycle is often long. These factors together pose a serious challenge to the stable operation of the braille embosser, forcing relevant agencies to increase manual identification efforts to deal with equipment failures, further exacerbating the pressure on manpower and costs.

[0004] In view of the above difficulties, introducing automated braille image recognition technology to identify and proofread the content of braille embossed products has become a key approach to solving the problem. However, despite the urgent need, the development of braille image recognition technology has been relatively slow. Traditional braille recognition research simplifies double-sided braille recognition into an independent single-sided recognition process for the front and back sides. This method requires the braille embossed product to be flipped and multiple images to be captured, which is not only inefficient but also limited by the physical structure of the braille embossing equipment in certain environments, making it impossible to perform the flipping operation. In addition, the recognition range is limited to braille dots, ignoring the information of braille dots. At the same time, the algorithm in the field of braille image recognition has made limited progress, and the current braille dataset resources are severely lacking. The existing public dataset mainly focuses on single-sided braille, and the annotation information only includes braille dot features, lacking information on concave dots. And most of the current braille image recognition methods still rely on traditional technology, the models used are outdated, and various braille datasets differ greatly in format and language. SUMMARY

[0005] To solve the above technical problems existing in the prior art, the application provides a double-sided Braille image recognition method and system based on a fusion language model and target detection.

[0006] In one aspect, to achieve the above object, the application provides a double-sided Braille image recognition method based on a fusion language model and target detection, comprising:

[0007] obtaining Braille image data to be recognized, wherein the Braille image data to be recognized is single-sided Braille image data containing front Braille information and back Braille information;

[0008] inputting single-sided Braille image data of any side into a double-sided Braille detection model for processing to obtain a double-sided detection result;

[0009] The double-sided Braille detection model is obtained by training a double-sided Braille image recognition data set, and the double-sided detection result includes the position and category information of front Braille convex points and back Braille concave points.

[0010] Preferably, the method further comprises correcting the double-sided detection result by a language model.

[0011] Preferably, the language model is a fine-tuned large language model, and the fine-tuned large language model is trained based on a self-defined Braille character-level Tokenizer.

[0012] Preferably, the double-sided Braille detection model comprises:

[0013] a dynamic feature aggregation module DFA for dynamically weighting features by adaptive aggregation and merging key features;

[0014] a feature enhancement and compression module FEC for enhancing Braille feature expression by three-layer cascaded convolution and performing feature compression;

[0015] a fusion improvement module SE for introducing global attention to improve the detection ability of Braille points.

[0016] Preferably, the processing process of the dynamic feature aggregation module DFA comprises:

[0017] channel attention weighting is performed on the main channel features divided by C2f to obtain weighted Braille features;

[0018] the weighted Braille features and secondary channel Braille features are spliced and input into a Bottleneck structure for deep Braille feature extraction;

[0019] Finally, all Braille features processed by the Bottleneck structure are spliced in the channel dimension to obtain output Braille features.

[0020] Specifically,

[0021]

[0022] X'2 = W·X2,

[0023] wherein, Conv 1 (1×1) is a first 1×1 convolution operation, ReLU is a nonlinear activation function, Conv 2 (1×1) is a second 1×1 convolution operation, σ(·) is a Sigmoid normalization, W is a weighted Braille point channel attention weight, X2 is a main channel Braille feature, and X'2 is a weighted Braille feature after DFA processing.

[0024] Preferably, the feature enhancement and compression module FEC is executed in the feature fusion stage, including:

[0025] The channel integration is performed on the Braille convex point and concave point features processed by the Bottleneck to obtain integrated Braille features, and the Braille convex point and concave point feature expression is refined to obtain refined Braille convex point and concave point features.

[0026] The original channel number is expanded, the Braille convex point and concave point feature expression capability is enriched, enhanced Braille convex point and concave point features are generated, and the enhanced Braille convex point and concave point features are compressed in channels.

[0027] Specifically,

[0028] Y = Conv (1×1) (Concat(Y1, Y2, …, Y n )),

[0029] Y' = Conv (3×3) (Y),

[0030] Y final = Conv (1×1) (Y3),

[0031] wherein, Y is the integrated Braille feature, Conv (1×1) is a 1×1 convolution operation, Y' is the refined Braille convex point and concave point feature, Conv (3×3) is a 3×3 convolution operation, Y3 is the enhanced Braille convex point and concave point feature map, and Y final is the feature after channel compression processing.

[0032] Preferably, constructing the double-sided Braille image recognition dataset includes:

[0033] The front-side Braille and the back-side Braille are labeled respectively using a labeling tool, and the labeled information is converted into a format recognizable by a deep learning model through a corresponding relationship;

[0034] The labeled content includes specific category information, specific content information, position information, and bounding box information of each Braille.

[0035] Preferably, the fine-tuned large language model comprises model-specific fine-tuning and model prediction fine-tuning.

[0036] The model-specific fine-tuning comprises:

[0037] A custom Braille character-level tokenizer is constructed, a dataset in the datasets library of Hugging Face is used for training, the batch size is set by combining the gradient accumulation strategy, the AdamW optimizer is used and weight decay is introduced to reduce the risk of overfitting;

[0038] The model prediction fine-tuning comprises:

[0039] The input Braille text is converted into a corresponding token sequence and input into the large language model, and the prediction logits of the last token are calculated; the logits are normalized by using the softmax function to convert into a probability distribution, and the top-k prediction results with the highest probability are selected; finally, the predicted token index is converted back to Braille characters by using the custom Braille character-level tokenizer.

[0040] Preferably, the double-side detection result is input into the fine-tuned large language model for correction, comprising a weighted comparison mode and a threshold comparison mode.

[0041] The weighted comparison mode is to calculate the weighted confidence of the two predictions according to adjustable weights, and to select the model output with the highest weighted confidence as the recognition result; the adjustable weights are YOLO weights and GPT weights.

[0042] The threshold comparison mode is to use the prediction result of the fine-tuned large language model as the recognition result when the YOLO confidence is lower than a set threshold and the large language model confidence is higher than a set threshold.

[0043] On the other hand, to achieve the above-mentioned purpose, the application further provides a double-side Braille image recognition system based on a fusion language model and target detection, comprising:

[0044] A data acquisition module is configured to acquire Braille image data to be recognized, wherein the Braille image data to be recognized is single-side Braille image data containing front-side Braille information and back-side Braille information.

[0045] The result output module is configured to input the single-sided Braille image data of any one side into the double-sided Braille detection model for processing to obtain a double-sided detection result.

[0046] The recognition correction module is configured to input the double-sided detection result into the fine-tuned large language model for correction to output a recognition result.

[0047] The double-sided Braille detection model is trained by a double-sided Braille image recognition dataset, and the double-sided detection result includes the position and category information of the convex points of the front side Braille and the concave points of the back side Braille.

[0048] Compared with the prior art, the present application has the following advantages and technical effects:

[0049] (1) The present application can obtain double-sided Braille information by photographing and recognizing the Braille image of any one side, avoiding the cumbersome process of flipping and multiple image acquisition in the traditional method, thereby greatly improving the recognition efficiency.

[0050] (2) The present application uses a new YOLO target detection model, which improves the recognition accuracy compared with the original target detection model, and combines the fine-tuned GPT-2 language model to correct the detection result, further improving the recognition accuracy, and can also adapt to different languages of Braille, enhancing the flexibility of target detection.

[0051] (3) The double-sided Braille detection model of the present application includes a dynamic feature aggregation module (DFA), a feature enhancement and compression module (FEC), and a fusion improvement module (SE), which optimizes the extraction and expression of Braille features through adaptive aggregation, feature enhancement and compression, and global attention introduction, and improves the detection capability of Braille points. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and their description are used to explain the application and are not intended to limit the application. In the drawings:

[0053] Figure 1 The figure is a structure diagram of the double-sided Braille detection model of the embodiment of the present application.

[0054] Figure 2 The figure is a structure diagram of C2f of the embodiment of the present application.

[0055] Figure 3 The figure is a comparison of the improved network setting and the original setting of the embodiment of the present application.

[0056] Figure 4 The figure is a structure diagram of DFA of the embodiment of the present application.

[0057] Figure 5 A structural diagram of an FEC module of an embodiment of the present application;

[0058] Figure 6 A module structural diagram of an SE of an embodiment of the present application;

[0059] Figure 7 A flow chart of double-sided Braille image recognition of an embodiment of the present application;

[0060] Figure 8 An AngelinaDataset front data set label information schematic diagram of an embodiment of the present application;

[0061] Figure 9 A double-sided Braille mapping relationship schematic diagram of an embodiment of the present application;

[0062] Figure 10 A running flow chart of a double-sided Braille image recognition system of an embodiment of the present application fusing a language model;

[0063] Figure 11 A comparison schematic diagram of a recognition method and a traditional method of an embodiment of the present application. DETAILED DESCRIPTION

[0064] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0065] It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although a logical order is shown in the flow chart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0066] The recognition process of double-sided Braille engraved products in the prior art is complicated, including taking a picture of one side and recognizing it, then turning over the engraved product and taking a picture again to recognize the content of the other side. This not only greatly reduces the work efficiency, but also may damage the Braille engraved product due to improper operation. More complicatedly, due to the physical structure characteristics of the Braille engraving machine and the limitations of the existing model, the Braille points as the front side to be recognized are often disturbed by the back side Braille points in the recognition process, thereby reducing the overall recognition accuracy and further increasing the artificial correction cost, forming a vicious cycle. In the face of these technical problems to be solved, the present embodiment proposes a breakthrough double-sided Braille image recognition method, which realizes the acquisition of the content information of both sides of the Braille engraved product through a single scanning. The comparison chart of the innovative double-sided Braille recognition method of the present embodiment and the traditional Braille recognition method is shown in Figure 11 .

[0067] As Figures 1-10 The embodiment provides a double-sided Braille image recognition method based on a fusion language model and target detection, and the method comprises the following steps:

[0068] obtaining Braille image data to be recognized, wherein the Braille image data to be recognized is single-sided Braille image data containing front Braille information and back Braille information;

[0069] inputting single-sided Braille image data of any side into a double-sided Braille detection model for processing to obtain a double-sided detection result;

[0070] The double-sided Braille detection model is obtained by training a double-sided Braille image recognition data set, and the double-sided detection result comprises position and category information of front Braille convex points and back Braille concave points.

[0071] Specifically, the method provided in the embodiment only needs to perform single shooting on any side of a Braille engraved product, and can recognize complete information of the front and back sides without page turning operation. The method fundamentally changes the technical path of Braille recognition, significantly reduces the labor and equipment costs, realizes a breakthrough in structural design, and greatly improves the model processing speed.

[0072] Further, the method further comprises correcting the double-sided detection result by using a language model.

[0073] Specifically, the language model is a fine-tuned large language model, which is a fine-tuned GPT-2 language model in the embodiment, and the fine-tuned GPT-2 language model is trained based on a self-defined Braille character-level Tokenizer.

[0074] Further, the double-sided Braille detection model comprises:

[0075] a dynamic feature aggregation module DFA, which is used for dynamically weighting features by adaptive aggregation and merging key features;

[0076] a feature enhancement and compression module FEC, which is used for enhancing Braille feature expression by three-layer cascaded convolution and performing feature compression;

[0077] a fusion improvement module SE, which is used for introducing global attention to improve the detection capability of Braille points.

[0078] Specifically, the embodiment proposes a double-side braille detection model DSBD (Double Side Braille Detection) capable of simultaneously identifying braille convex points and concave points based on braille features, and designs three independent improvement modules for specific challenges of braille detection: DFA, FEC, and an improvement module of the fusion SE module. The DFA module dynamically weights the features through adaptive aggregation, and merges the key features; the FEC module enhances the detection capability of braille features through feature enhancement, and reduces the calculation amount and improves the efficiency through feature compression; the improvement module of the fusion SE module improves the detection capability of braille points by introducing global attention. The model structure diagram is as shown in Figure 1 The C3K2-Braille (C3K2-B) part contains the C2f structure of the integrated innovative module, and the model discards the C2PSA module which will cause the training efficiency to decrease due to internal operation. The three modules are independent optimization schemes, which are integrated into the C2f structure to train the model in the experiment. The structure diagram of C2f is as shown in Figure 2 .

[0079] The double-side braille detection model includes but is not limited to the Vision Transformer model, the YOLO model, and the RetinaNet model.

[0080] The embodiment adjusts the original network settings of the YOLOv11 target detection model to adapt to the particularity of the braille dataset.

[0081] First, in terms of the number of categories, the default setting of the original YOLO11 is 80 categories of targets, while the braille dataset of the embodiment contains at most 128 categories of targets. Therefore, the nc parameter is adjusted to 128 to ensure that the model can cover all braille character categories.

[0082] Second, in terms of model scale, the multiple scales (n, s, m, l, x) of the original model are optimized in terms of computational complexity and parameter amount for general object detection, but are not suitable for the braille dataset. The embodiment selects the l scale (i.e. depth=1.00, width=1.00, and the maximum channel number is 1024) to ensure that the remaining settings will not be affected by the parameters, while avoiding the problems of high computational cost and limited maximum channel number.

[0083] In the backbone network design, the number of convolution channels in each layer is adjusted to be more suitable for Braille detection. The original YOLO11 uses a larger number of channels, such as a progressive structure of 256→512→1024, while the embodiment adjusts it to 48→128→192→384→576, reducing the computational overhead while maintaining sufficient feature expression capability. In addition, in order to enhance the capture ability of Braille, the number of repetitions of some C3k2 modules is increased. The first and second C3K2 layers pay more attention to the details of low-level features and target differentiation, using False to avoid excessive computational complexity. The third and fourth C3K2 layers deal with higher-level features. As the resolution of the feature map decreases, the target becomes more blurred and possibly more dense. At this time, using the True setting can effectively enhance the context awareness, helping the network to better distinguish and identify the target. Thus, the local features of Braille are extracted more deeply.

[0084] In the network configuration, the first C3K2 layer is set to True: This is the first layer of the network, which mainly processes the low-level features of the input image. Since the resolution of this layer of feature map is high, and Braille has not been fully downsampled, it can be set to True to enhance the network's ability to capture the details of Braille. The second C3K2 layer is set to True: The size of the feature map of this layer has been reduced, but it may still contain some relevant information about Braille, especially after the later convolution operation. Setting True can enhance the recognition ability of this layer for these Braille. The last few C3K2 layers remain False: As the network enters deeper levels, the resolution of the feature map has been reduced very low. Using traditional convolution methods may be more beneficial to reduce the computational burden without losing the perception of global information. At this time, the details of Braille have been gradually captured, so there is no need to pay excessive attention. The improved network setting is compared with the original setting as shown in Figure 3 .

[0085] Further, the processing process of the dynamic feature aggregation module DFA includes:

[0086] The main channel features divided from C2f are subjected to channel attention weighting to obtain weighted Braille features;

[0087] The weighted Braille features are spliced with the secondary channel Braille features and input into the Bottleneck structure for deep Braille feature extraction;

[0088] Finally, all Braille features processed by the Bottleneck structure are spliced in the channel dimension to obtain output Braille features;

[0089] Specifically, the embodiment introduces a dynamic feature integration (DFA) module in the C2f module. After feature extraction of each Braille image, the most relevant features are selected for merging by a dynamic aggregation module based on the importance and feature similarity of each part in the Braille image. The structure of DFA is shown in Figure 4

[0090] DFA performs feature weighting on the main channel Braille features X2 divided by C2f through a lightweight attention mechanism to highlight Braille point features and suppress redundant information. The calculation process is as follows:

[0091]

[0092] where Conv 1 (1×1) dimensionality reduction to extract global Braille information, ReLU as a nonlinear activation function, Conv 2 (1×1) Braille convex point and concave point channel weight mapping, and σ(·) represents Sigmoid normalization to generate weighted Braille point channel attention weights W. Finally, the DFA module adjusts the feature expression of the main channel Braille features X2 through weighted calculation:

[0093] X′2=W·X2,

[0094] After DFA processing, the weighted Braille features X′2 are spliced with the secondary channel Braille features X1 and input into multiple Bottleneck structures for deep Braille feature extraction. Finally, all the Braille features processed by Bottleneck are spliced in the channel dimension to obtain the output Braille features Y.

[0095] Compared with the standard C2f module, the introduction of the DFA mechanism improves the model performance in several key aspects. First, this mechanism gives the model stronger adaptive feature selection capability, which can dynamically adjust the focus area according to the input image content, so that the model can accurately focus on the key features of Braille convex points and concave points, while effectively suppressing the interference of irrelevant information such as blank areas.

[0096] Second, the dynamic focusing mechanism introduced by the DFA module solves the limitations of the fixed processing mode in Braille recognition. Traditional methods usually apply the same feature extraction strategy to all regions, while DFA can adaptively allocate computing resources to regions containing Braille information, greatly improving the model's ability to process overlapping and dense Braille points.

[0097] Assuming that the input Braille feature map size is H×W, the channel number is C, and the calculation complexity of the standard Bottleneck structure is mainly determined by the 3×3 convolution operation, the calculation amount is:

[0098] O​Bottleneck = O(n·C 2 ·H·W),

[0099] wherein n is the number of Bottleneck layers. The DFA only introduces two 1x1 convolutions, and the additional computational cost is:

[0100] O DFA = O(C 2 ·H·W),

[0101] Since the computational cost of 1x1 convolution is much lower than that of 3x3 convolution, the computational cost of DFA is relatively low, but it can significantly improve the feature expression ability of Braille convex and concave points.

[0102] Further, the feature enhancement and compression module FEC is executed in the feature fusion stage, including:

[0103] integrating the channels of the Braille convex and concave point features processed by the Bottleneck to obtain integrated Braille features, and refining the Braille convex and concave point feature expression to obtain refined Braille convex and concave point features;

[0104] expanding the original channel number, enriching the Braille convex and concave point feature expression ability, generating enhanced Braille convex and concave point features, and compressing the channels of the enhanced Braille convex and concave point features;

[0105] Specifically, the feature enhancement and compression module FEC enhances the feature expression ability through multi-layer feature enhancement, and optimizes the computational complexity through channel compression mechanism, so that the model can effectively extract the features of small targets while maintaining the computing efficiency, improve the detection accuracy, and the structure diagram of the FEC module is as shown in Figure 5 .

[0106] In the feature fusion stage of C2f, the Braille convex and concave point features processed by the Bottleneck are first integrated through 1x1 convolution to obtain integrated Braille features Y:

[0107] Y = Conv (1×1) (Concat(Y1, Y2, …, Y n )),

[0108] Then, 3x3 convolution is used to refine the Braille convex and concave point feature expression to obtain refined Braille convex and concave point features Y':

[0109] Y' = Conv (3×3) (Y),

[0110] In the multi-layer feature enhancement stage, three additional 3x3 convolution layers are introduced to enrich the Braille convex and concave point feature expression ability:

[0111]

[0112] Finally, to reduce the computational complexity and improve feature compactness, the enhanced Braille convex and concave feature map Y3 is compressed by 1x1 convolution:

[0113] Y final = Conv(1x1)(Y3),

[0114] The core design concept of the FEC module is the "expansion-purification-compression" strategy of feature dimension. The input feature channel is expanded to twice the original through the initial convolution layer (enhance conv1), then further expanded to four times through the intermediate layer (enhance conv2), and finally compressed through (enhance conv3) to form a highly discriminative feature representation. This channel expansion strategy significantly enhances the model's ability to perceive weak features of Braille points, effectively solving the problem of adjacent point interference in Braille detection. Especially in high-density Braille documents, the enhanced feature representation can effectively distinguish closely distributed Braille convex and concave points, reducing the false detection rate caused by spatial distribution interference.

[0115] To address the challenges of varying lighting conditions, the FEC module provides a larger receptive field and stronger feature expression capability through a three-layer 3x3 convolution cascade design, enabling the network to capture discriminative features of Braille points under different lighting conditions from multiple scales. This multi-level feature extraction mechanism improves the model's adaptability to lighting changes and reduces the problem of convex point features being weakened under strong light or concave point misidentification under weak light.

[0116] In terms of material variation, the last layer (compress conv) of the FEC module uses 1x1 convolution for feature compression, effectively removing redundant information and retaining key discriminative features in material variation. This design balances computational efficiency and feature expression capability, enabling the model to maintain high accuracy while handling Braille carriers of different materials.

[0117] The main computational load of FEC comes from multi-layer feature enhancement and feature compression. Due to the addition of three layers of 3x3 convolution, the FEC structure slightly increases the computational load compared to the original C2f. The computational complexity of feature compression can be represented as:

[0118] O Compress = O(C 2 · H· W),

[0119] Although this method slightly increases the computational cost, the growth rate of computational complexity is relatively limited, while the feature expression capability is significantly enhanced.

[0120] Further, the SE (Squeeze-and-Excitation) module is a channel attention mechanism mainly used to improve the feature expression capability. The core idea is to first extract the global information of each channel through global average pooling, then model the dependence relationship between channels by using two-layer fully connected network, finally generate channel weights, and reweight the original features to enhance the expression of key features and suppress irrelevant information. However, there is currently no special research on the fusion of SE module and C2f structure in the field of Braille image recognition. Since C2f mainly relies on local feature fusion, it lacks explicit global channel information modeling, and when facing small size Braille dots, concave and convex points and densely overlapping Braille, it may not be able to fully utilize global features. The introduction of SE module can make up for this deficiency. Through the channel attention mechanism, SE module can dynamically adjust the weights of different channels, so that the Braille features extracted by C2f pay more attention to key Braille information with discriminative power while fusing multi-scale features, thereby improving the detection accuracy of small size Braille concave and convex points. In addition, in the recognition of overlapping Braille, SE module can effectively enhance the discriminability of Braille features, help the model to separate the foreground target more clearly, and reduce the influence of feature aliasing. The structure diagram of SE module is shown in Figure 6

[0121] In the process of Bottleneck calculation, multi-scale feature fusion (MSF) is introduced, and different size convolution kernels are used to extract Braille point information of different scales to enhance the recognition ability of the network to small size Braille points and multi-scale Braille:

[0122] Y MSF =Conv (3×3) (Y)+Conv (5×5) (Y)+Conv (7×7) (Y),

[0123] On this basis, SE mechanism is used to calculate the global channel weight to improve the stability of Braille point feature expression. First, the global Braille feature Z is calculated and obtained through global average pooling (GAP):

[0124] Z=GAP(Y MSF ),

[0125] Then, 1×1 convolution is used to reduce the dimension of the channel to obtain the reduced dimension Braille feature Z':

[0126]

[0127] Then restore the channel dimension by 1×1 convolution and use Sigmoid normalization:

[0128] ​

[0129] Finally, the calculated channel weight W acts on the input feature to enhance the key Braille point feature:

[0130] Y SE =W·Y MSF ,

[0131] After the Bottleneck calculation and attention mechanism processing, the Braille point feature is fused through 1×1 convolution to generate the final output:

[0132] Y final =Conv (1×1) (Concat(Y SE ,Y MSF )),

[0133] Compared with the standard C2f structure, the SE module has several unique advantages. By introducing different scale feature extraction paths through multi-scale feature fusion (MSF), the feature expression of small size Braille points is more rich, which significantly improves the adaptability to Braille of various scales. In terms of feature expression stability, the SE module can dynamically adjust the feature weight under different input data distribution, making the Braille point feature expression more stable and enhancing the generalization ability of the model. This fusion method not only makes up for the lack of global channel information modeling of C2f structure, but also realizes the balance between high efficiency and low computational overhead.

[0134] The computational complexity of this module mainly comes from the Bottleneck structure, multi-scale feature fusion (MSF) and SE mechanism.

[0135] The Bottleneck calculation complexity can be represented as:

[0136] O Bottleneck =O(n·C 2 ·H·W),

[0137] Where n is the number of Bottleneck layers. Multi-scale feature fusion (MSF) uses 3×3, 5×5 and 7×7 convolution kernels, and its computational complexity is:

[0138] O MSF =O(C 2 ·H·W)+O(C 2 ·H·W)+O(C 2 ·H·W)=O(3·C 2 ·H·W),

[0139] The computational complexity of SE mechanism is mainly determined by 1×1 convolution and global pooling operation:

[0140] O SE = O(n·C 2 ·H·W),

[0141] Finally, the total computational complexity of this structure is:

[0142] O Total = O(n·C 2 ·H·W) + O(3·C 2 ·H·W) + O(C 2 ·H·W) = O((n+4)·C 2 ·H·W),

[0143] O((n+4)·C 2 ·H·W),

[0144] It can be seen that the growth of the method in computational complexity is relatively limited, while the target detection performance can be effectively improved.

[0145] Further, constructing the double-sided Braille image recognition dataset comprises:

[0146] Using a labeling tool to label the front and back Braille respectively, and converting the labeled information into a format that can be recognized by a deep learning model through a corresponding relationship;

[0147] Wherein, the labeled content includes the specific category information, specific content information, position information, and detection box information of each Braille.

[0148] Specifically, based on the analysis of the public dataset, LabelMe is selected as the tool for Braille image labeling. In order to ensure the labeling quality, a detailed labeling specification and process guide is developed, covering the recognition standard and labeling points of back Braille. Subsequently, a number of students are carefully selected and trained as research assistants to ensure that they fully understand the labeling specification and master the operation skills of LabelMe tool. In the actual labeling process, the team jointly completes the back Braille labeling work of 290 Braille images, which have high complexity, and each image contains about 300 back Braille units on average. Due to the subtle features and large number of back Braille, the labeling of a single image usually requires several hours of focused work, and the entire project lasts for fourteen days of intensive labor.

[0149] In order to maximize the accuracy and consistency of the dataset, this embodiment implements a strict double verification mechanism: first, all initial labeling results are proofread for the first time to identify errors and inconsistencies; then a second round of fine correction work is carried out to focus on solving the problem points found before. The final completed dataset contains accurate back Braille position and category information.

[0150] DSBI(Double-Sided Braille Image Dataset), which is a double-sided braille image dataset for algorithm evaluation of braille dot (DOTS) detection. The dataset contains 114 color double-sided braille images from six different braille books and six separately printed braille documents. These images are all acquired by a flatbed scanner to ensure image quality and facilitate data collection. The dataset uses a HP LaserJet Pro MFP M226DN scanner and adopts a 200DPI resolution for color image acquisition, finally stored in JPEG format to reduce storage occupancy while ensuring image clarity.

[0151] For each double-sided braille image, the dataset provides corresponding braille dot annotations, including braille recto dot (file suffix "+recto.txt") and braille verso dot (file suffix "+verso.txt"). The annotation file records the skew angle, vertical line position, horizontal line position, and braille cell information of the braille dot in detail.

[0152] Among them, the first row of skew angle indicates the rotation direction of the braille dot, and the positive value represents clockwise rotation, and the negative value represents counterclockwise rotation. The second row of vertical line position is used to determine the column structure of the braille cell, and its number must be even, and the third row of horizontal line position determines the row structure, and its number must be a multiple of 3. In the specific braille cell annotation part, each row contains eight numbers, where the first two numbers represent the row number and column number (index starts from 1) of the braille cell, and the last six numbers are used to represent the structure of the braille dot, where "1" represents the braille dot and "0" represents the background.

[0153] In addition, the dataset also provides images related to braille recto dots (file suffix "+recto.jpg") and braille verso dots after skew correction (file suffix "+verso.jpg"). In order to facilitate researchers to process and optimize the data, the dataset also provides related functions implemented in C++ and OpenCV. For example, for the original image with skew angle, the image can be de-skewed by calling the function, and the corrected image can be obtained. This function helps to improve the recognition accuracy of braille dots and ensures the consistency of data processing. The data used in this embodiment are all de-skewed pictures and label information.

[0154] Processing method of DSBI dataset:

[0155] The dataset contains braille information on both sides, so it needs to be converted to YOLO format when used to adapt to the braille detection task. The conversion process mainly includes data reading, annotation parsing, coordinate normalization and data storage, and the specific steps are as follows.

[0156] First, the code reads the Braille category map file, initializes the input and output paths, and creates a directory for storing YOLO annotation files. The program then traverses the dataset directory and its subdirectories, searching for matching Braille annotation files, including all front and back label files, and matches them to the corresponding image files to obtain the image width (W) and height (H).

[0157] In the process of parsing the annotation file, the code extracts the following key data: vertical line position set V = {v1, v2, ..., v n}, where n must be an even number and is used to determine the column structure of the Braille cells.

[0158] Horizontal line position set H = {h1, h2, ..., h m}, where m must be a multiple of 3 and is used to determine the row structure of the Braille cells.

[0159] Braille unit data, including row number (r), column number (c) and six binary Braille dot representation.

[0160] During the braille dot category mapping process, the code converts the binary representation of each braille cell into a category number C, namely:

[0161] C=braille\_map(binary)#(3-22),

[0162] If the code is not in the mapping table, the Braille cell is skipped to ensure that only valid categories of data are stored.

[0163] Next, the code calculates the normalized coordinates in YOLO format, namely the center coordinates (x_center, y_center) and bounding box size (w, h) of each Braille cell, and normalizes them to between [0, 1]:

[0164]

[0165] Among them, v and h correspond to the positions of the vertical and horizontal lines respectively.

[0166] When calculating the width (w) and height (h) of the bounding box, the code fully considers the diameter d of the Braille dot, ensuring that the bounding box is slightly larger than the actual Braille dot to improve the robustness of the detection:

[0167]

[0168] Finally, the calculated YOLO format data is stored in the following format:

[0169] C x center y center wh.

[0170] AngelinaDataset dataset:

[0171] This dataset contains annotated Braille text photos and covers Braille samples from various sources to support Braille character recognition and related deep learning tasks. The dataset mainly includes the following:

[0172] First, the dataset consists of 212 pages of double-sided Braille books, 28 student handwritten Braille works, and an additional 50-page test set, ensuring data diversity and wide applicability. This dataset only contains frontal data sets. In terms of annotation format, each line represents the bounding box of a Braille character and its corresponding character label, in the following format:

[0173] <left> ； <top> ； <right> ； <bottom> ； <label>,

[0174] wherein left, top, right, bottom represent the coordinates of the character bounding box, normalized to the range [0, 1], (0, 0) represents the top left corner of the image, (1, 1) represents the right bottom corner. label is the integer label of the Braille character, taking the value range [1, 63], used to identify the blind spots contained in the character. Among them, the 6 blind spots are coded in binary, the 1st point corresponds to the lowest bit, and the 6th point corresponds to the highest bit.

[0175] Processing method of AngelinaDataset dataset:

[0176] Firstly, the definition of Braille in this dataset is different from the standard Braille specification. The standard Braille specification has a label of 100100 for 001001, so a mapping table needs to be added in the label conversion process to make the annotation information of AngelinaDataset dataset the same format as the standard Braille specification. Secondly, this dataset only contains front Braille information, so the front Braille data is directly processed. This dataset provides labels in json format and csv format. The storage format of the data in the csv format dataset is <left> ； <top> ； <right> ； <bottom> ； <label>Therefore, the self-developed code is used here to convert the csv annotation information file of the positive dataset into a YOLO format txt file, as follows:

[0177] C x center y center wh,

[0178] The label information of the positive dataset is as shown in Figure 8 The negative Braille data is not given, so the Labelme annotation tool is used for annotation in this embodiment, and the annotation standard of the label is consistent with the standard Braille specification.

[0179] The program first traverses the specified JSON directory and reads the annotation files one by one. Each JSON file corresponds to an image and stores the information of all Braille characters in the image. In order to ensure the accuracy of the conversion, the program matches the corresponding image according to the JSON file name and obtains its width (W) and height (H). If the image is missing, skip the processing of this file to ensure data integrity.

[0180] When parsing the JSON annotation data, the program reads the shapes field, which contains the bounding box coordinates and character labels of all Braille characters. The label of the Braille character uses 6-bit binary coding, such as "110001", where 1 represents the presence of a Braille dot and 0 represents the absence of a Braille dot. In order to match the class format of YOLO training, the binary coding needs to be converted to a decimal class index, and the conversion formula is:

[0181]

[0182] where b i represents the value of the i-th Braille dot (0 or 1), and the calculated C value ranges between [0, 63]. Subsequently, the program extracts the points field in the JSON file, which provides the bounding box information of the Braille character, i.e. the left upper corner coordinates (x1, y1) and the right lower corner coordinates (x2, y2).

[0183] In order to adapt to the YOLO format, the program normalizes the bounding box coordinates so that their values are limited to [0, 1]. The normalization calculation formula is as follows:

[0184]

[0185] After normalization, the program stores the annotation data of each Braille character according to the YOLO format, with the following format:

[0186] C x center y center wh,

[0187] where C represents the class index, x center and y center represent normalized center point coordinates, w and h are the normalized width and height of the bounding box, respectively. The program then stores the converted data as a.txt file and places it in the designated output directory. If the JSON file does not contain any valid Braille characters, an empty.txt file is generated to ensure the integrity of the annotated data. Finally, all JSON annotation files are converted one by one.

[0188] After processing, the dataset is further divided into 9 sub-datasets, covering DSBI front, DSBI back, DSBI double (back label + 64), AngelinaDataset front, AngelinaDataset back, AngelinaDataset double (back label + 64), and the front, back, double (back label + 64) of the merged dataset (DSBI + Angelina). This data division method ensures the compatibility of different data sources and provides flexible configuration for subsequent experiments. At this point, all processing procedures for the self-made double-sided Braille dataset have been completed.

[0189] After obtaining the processed dataset, the dataset is divided into training set and validation set, and the improved YOLO model is used for training. In the training process, a specially processed mapping relationship is adopted: the front Braille uses the standard binary to decimal mapping relationship, and the back Braille uses the mapping relationship based on the standard binary to decimal mapping relationship with a suffix, such as Figure 9 , so that the model can distinguish the class information of front and back Braille, and obtain a model that can recognize front and back Braille at the same time.

[0190] Further, the fine-tuned large language model includes model-specific fine-tuning and model prediction fine-tuning;

[0191] Among them, the model-specific fine-tuning includes:

[0192] A custom Braille character-level Tokenizer is constructed, trained with datasets from Hugging Face's datasets library, and combined with a gradient accumulation strategy to set the batch size, using the AdamW optimizer and introducing weight decay to reduce the risk of overfitting;

[0193] Specifically, this embodiment adopts a generative pre-trained Transformer-2 (GPT-2) model to fine-tune the Braille text in depth to improve its performance in the Braille language modeling task. GPT-2 is an autoregressive language model based on the Transformer architecture, with strong text generation capabilities. However, the standard GPT-2 model mainly relies on large-scale natural language text data during training, and its modeling capability for Braille text is still insufficient. Therefore, this embodiment is based on the pre-trained GPT-2, combined with a custom Braille character-level Tokenizer and an optimized training strategy, to enhance its adaptability to Braille data.

[0194] First, this embodiment loads a custom Braille character-level Tokenizer that has been specifically trained, and ensures that its pad token is consistent with the GPT-2 eos token to avoid errors that may occur during padding. Subsequently, the embedding layer of the GPT-2 model is adjusted to adapt to the new Braille vocabulary, ensuring that the model can correctly process Braille input.

[0195] In terms of data processing, the datasets library of Hugging Face is used to efficiently load and process text data. The training dataset is preprocessed by standardization, including text tokenization, truncation, and padding, to ensure that all samples have a fixed length, thereby improving the learning stability of the model. In addition, the dataset is converted to PyTorch format to meet the input requirements of the GPT-2 model.

[0196] To optimize the Braille text generation model, a series of systematic fine-tuning strategies are adopted. In terms of training parameter configuration, the batch size is set to 16 and combined with a gradient accumulation strategy (set to 4-step accumulation) to improve GPU utilization while effectively reducing memory pressure. The learning rate parameter is set to 2e-5 with a warm-up period of 1000 steps to ensure stability in the early stages of training and high-quality convergence in the final stage. In addition, the AdamW optimizer is used and weight decay is introduced to reduce the risk of overfitting, while enabling mixed precision training further optimizes the utilization of computing resources, significantly improving the training efficiency of large-scale Braille datasets. During training, the model saves a checkpoint every 5000 steps and logs every 1000 steps to track the training progress. The training process uses 5 training cycles, combined with an evaluation dataset for regular evaluation to ensure that the model can steadily converge during training.

[0197] Model prediction fine-tuning includes:

[0198] The input Braille text is converted into a corresponding token sequence and input into the GPT-2 model to calculate the prediction logits of the last token. The logits are normalized by using the softmax function to convert them into a probability distribution, and the top-k prediction results with the highest probability are selected. Finally, the predicted token index is converted back to Braille characters using the self-defined Braille character-level Tokenizer.

[0199] Specifically, the fine-tuned GPT-2 model is used to predict the Braille text to evaluate the generalization ability of the model in the Braille text modeling task. As an autoregressive language model, GPT-2 can generate the most likely subsequent text based on the given context. However, the traditional GPT-2 model lacks training for Braille and cannot effectively process Braille character sequences. This embodiment loads the self-defined Braille character-level Tokenizer and the deep fine-tuned GPT-2 model, and uses probability statistics method to predict the most likely subsequent characters for a given Braille input.

[0200] During the experiment, the model first converts the input Braille text into a corresponding token sequence and inputs it into the GPT-2 model to calculate the prediction logits of the last token. Then, the logits are normalized by using the softmax function to convert them into a probability distribution, and the top-k prediction results with the highest probability are selected. Finally, the predicted token index is converted back to Braille characters using the self-defined Braille character-level Tokenizer for visual analysis.

[0201] Compared with traditional prediction methods, this embodiment realizes multi-aspect systematic optimization. In the text input link, the general natural language Tokenizer that does not support Braille is abandoned, and a self-defined Braille character-level Tokenizer is used instead, which significantly improves the accuracy of Braille character parsing and effectively avoids the problem of mis-splitting. At the model level, the standard GPT-2 is specially fine-tuned to adapt to the characteristics of Braille data, enhancing the model's understanding ability and prediction accuracy for Braille. In the output strategy, the limitations of traditional greedy search are broken through, and the Top-K sampling mechanism is introduced to improve the diversity and context coherence of the prediction results. In the probability calculation aspect, the softmax normalization is used instead of the simple maximum logits value calculation to generate a more interpretable probability distribution, avoiding the problem of prediction results being excessively biased towards high-frequency characters. In addition, by maintaining the autoregressive prediction mechanism while optimizing the Token parsing process, the correct output of Braille characters is ensured, significantly improving the performance and applicability of the model in the Braille prediction task.

[0202] Further, the detection result is input into a fine-tuned GPT-2 language model for correction, including a weighted comparison mode and a threshold comparison mode.

[0203] The weighted comparison mode is to calculate the weighted confidence of the two predictions according to adjustable weights, and select the model output with the highest weighted confidence as the recognition result; the adjustable weights are YOLO weights and GPT weights.

[0204] The threshold comparison mode is to use the prediction result of GPT-2 as the recognition result when the YOLO confidence is lower than the set threshold and the GPT-2 confidence is higher than the set threshold.

[0205] Specifically, the embodiment designs and implements a comprehensive dual-blind image recognition multi-model fusion architecture, effectively solving the key challenges in traditional blind recognition. The architecture consists of two core modules: a YOLO-based visual recognition module and a GPT-2-based language prediction module.

[0206] At the visual recognition level, the front and back combined single model dual-blind image recognition system deploys a specially trained YOLO model that can simultaneously optimize both sides of the blind text, and introduces a language model after the recognition work, as shown in Figure 10 The language model can predict the most likely blind characters and their corresponding prediction confidence at a specific position based on the context relationship of Chinese blind text. When the recognition confidence of the YOLO model is insufficient, the GPT-2 language model will predict the most likely blind characters based on context semantic analysis. Two prediction fusion modes are supported: weighted comparison mode and threshold comparison mode. In the weighted comparison mode, the system calculates the weighted confidence of the two predictions according to adjustable weights (YOLO weights and GPT weights), and selects the highest one; in the threshold comparison mode, only when the YOLO confidence is lower than the set threshold and the GPT-2 confidence is higher than the set threshold, the prediction result of GPT-2 is used.

[0207] In this embodiment, a complete decision process recording function is implemented, which saves detailed information for each recognized character, including original YOLO prediction, GPT-2 prediction, final decision and decision reason, facilitating subsequent analysis and system optimization. The recognition result is processed by a special conversion function to ensure correct output of blind Unicode characters.

[0208] This embodiment is based on YOLO visual recognition and introduces GPT-2 language model as an auxiliary decision mechanism, realizing two prediction fusion modes: weighted comparison mode and threshold comparison mode. When the YOLO model recognition confidence is insufficient, it can generate reasonable blind character prediction based on language context, effectively improving the recognition accuracy.

[0209] The embodiment also provides a double-sided Braille image recognition system based on a fusion language model and target detection, comprising:

[0210] a data acquisition module configured to acquire Braille image data to be recognized, wherein the Braille image data to be recognized is single-sided Braille image data containing front Braille information and back Braille information;

[0211] a result output module configured to input single-sided Braille image data of any side into a double-sided Braille detection model for processing to obtain a double-sided detection result;

[0212] a recognition correction module configured to input the double-sided detection result into a fine-tuned large language model for correction to output a recognition result;

[0213] The double-sided Braille detection model is obtained by training a double-sided Braille image recognition data set, and the double-sided detection result includes position and category information of front Braille convex points and back Braille concave points.

[0214] The above is only a preferred specific embodiment of the application, but the protection scope of the application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.< / label> < / bottom> < / right> < / top> < / left> < / label> < / bottom> < / right> < / top> < / left>

Claims

1. A double-sided Braille image recognition method based on fusion language model and object detection, characterized in that: include: Acquire Braille image data to be recognized, wherein the Braille image data to be recognized is single-sided Braille image data containing front-side Braille information and back-side Braille information; Input the single-sided Braille image data of any side into the double-sided Braille detection model for processing to obtain the double-sided detection result; The double-sided Braille detection model is obtained by training a double-sided Braille image recognition dataset, and the double-sided detection result includes the position and category information of the convex dots of the front-side Braille and the concave dots of the back-side Braille.

2. The double-sided Braille image recognition method based on fusion language model and target detection according to claim 1, characterized in that: The method further includes correcting the two-face detection result by using a language model.

3. The double-sided Braille image recognition method based on fusion language model and target detection according to claim 2, characterized in that: The language model is a fine-tuned large language model, and the fine-tuned large language model is trained based on a custom Braille character-level Tokenizer.

4. The double-sided Braille image recognition method based on fusion language model and target detection according to claim 1, characterized in that: The double-sided Braille detection model includes: Dynamic feature aggregation module DFA: used to dynamically weight features through adaptive aggregation and merge key features; Feature Enhancement and Compression Module (FEC): used to enhance Braille feature expression and perform feature compression through three layers of cascaded convolution; Fusion improvement module SE: used to introduce global attention to improve the detection ability of Braille dots.

5. The double-sided Braille image recognition method based on fusion language model and target detection according to claim 4, characterized in that: The processing process of the dynamic feature aggregation module DFA includes: Perform channel attention weighting on the main channel features divided by C2f to obtain weighted Braille features; The weighted Braille features and the secondary channel Braille features are spliced ​​and input into the Bottleneck structure to perform deep Braille feature extraction; Finally, all the Braille features processed by the Bottleneck structure are spliced ​​in the channel dimension to obtain the output Braille features; Specifically: X′2=W·X2, Where, Conv 1 (1×1) is the first 1×1 convolution operation, ReLU is the nonlinear activation function, Conv 2 (1×1) is the second 1×1 convolution operation, σ(·) is the Sigmoid normalization, W is the weighted Braille dot channel attention weight, X2 is the main channel Braille feature, and X′2 is the weighted Braille feature after DFA processing.

6. The double-sided Braille image recognition method based on fusion language model and target detection according to claim 4, characterized in that: The feature enhancement and compression module FEC is executed in the feature fusion stage, including: Perform channel integration on the Braille convex and concave features processed by Bottleneck to obtain integrated Braille features, and refine the Braille convex and concave feature expressions to obtain refined Braille convex and concave features; Expanding the number of original channels, enriching the expression capability of Braille convex and concave features, generating enhanced Braille convex and concave features, and performing channel compression on the enhanced Braille convex and concave features; Specifically: Y=Conv (1×1) (Concat(Y1,Y2,…,Y n )), Y′=Conv (3×3) (AND), Y final =Conv (1×1) (Y3), Where Y is the integrated Braille feature, Conv (1×1) is a 1×1 convolution operation, Y′ is the refined Braille convex and concave features, Conv (3×3) is a 3×3 convolution operation, Y3 is the enhanced Braille convex and concave feature map, Y final It is the feature after channel compression processing.

7. The double-sided Braille image recognition method based on fusion language model and target detection according to claim 1, characterized in that: Constructing the double-sided Braille image recognition dataset includes: Use annotation tools to annotate the front and back Braille texts separately, and convert the annotated information into a format that can be recognized by the deep learning model through correspondence. The annotated content includes the specific category information, specific content information, location information, and detection frame information of each Braille character.

8. The double-sided Braille image recognition method based on fusion language model and target detection according to claim 3, characterized in that: The fine-tuning of the large language model includes model-targeted fine-tuning and model-prediction fine-tuning; The targeted fine-tuning of the model includes: We built a custom Braille character-level tokenizer, trained on datasets from Hugging Face's datasets library, set the batch size using a gradient accumulation strategy, and used the AdamW optimizer with weight decay to reduce overfitting risk. The model prediction fine-tuning includes: The input Braille text is converted into a corresponding token sequence and fed into the large language model to calculate the predicted logits of the last token. The logits are normalized using the softmax function, converted into a probability distribution, and the top-k prediction results with the highest probability are selected. Finally, a custom Braille character-level Tokenizer is used to convert the predicted token index back into Braille characters.

9. The double-sided Braille image recognition method based on fusion language model and target detection according to claim 3, characterized in that: Inputting the two-sided detection result into the fine-tuned large language model for correction, including: a weighted comparison mode and a threshold comparison mode; The weighted comparison mode is as follows: the weighted confidence of the two predictions is calculated according to the adjustable weights, and the model output with the highest weighted confidence is selected as the recognition result; the adjustable weights are YOLO weights and GPT weights; The threshold comparison mode is: when the YOLO confidence is lower than the set threshold and the large language model confidence is higher than the set threshold, the prediction result of the fine-tuned large language model is used as the recognition result.

10. A double-sided Braille image recognition system based on fusion language model and object detection, characterized in that: include: A data acquisition module is used to acquire Braille image data to be recognized, wherein the Braille image data to be recognized is single-sided Braille image data containing front-side Braille information and back-side Braille information; Result output module: used to input the single-sided Braille image data of any side into the double-sided Braille detection model for processing to obtain the double-sided detection results; Recognition and correction module: used to input the two-face detection result into the fine-tuned large language model for correction and output the recognition result; The double-sided Braille detection model is obtained by training a double-sided Braille image recognition dataset, and the double-sided detection result includes the position and category information of the convex dots of the front-side Braille and the concave dots of the back-side Braille.

Images