Braille character detection and recognition method and system based on Braille dot-level semantic enhancement

Abstract

A Braille character detection and recognition method and system based on Braille dot-level semantic enhancement includes: obtaining the location and category confidence of Braille characters in an image using computer vision technology; obtaining the salient point confidence of image pixels using computer vision technology; and screening Braille characters based on the Braille character category confidence and the salient point confidence within a region. This invention improves the existing problem of inaccurately identifying ambiguous Braille dots when using target detection models for Braille character detection and recognition, thereby improving detection and recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a Braille character detection and recognition method and system based on Braille dot-level semantic enhancement. Background Art

[0002] Braille translation can help sighted individuals without any prior knowledge of Braille read Braille. It can also improve the efficiency of Braille teachers, promoting the development of Braille education and expanding educational opportunities for the blind. Braille character detection and recognition is the first step in the translation process. Existing technologies cannot accurately recognize fuzzy Braille, ultimately resulting in inaccurate translation results.

[0003] Existing solutions involve expanding the dataset used for model training to improve model accuracy and minimizing noise in images, such as by using scanning devices to obtain high-quality images. However, creating Braille datasets is difficult and expensive. Using scanning devices to capture images is also inconvenient, limiting the scope of this approach. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a Braille character detection and recognition method and system based on Braille dot-level semantic enhancement.

[0005] To solve the above technical problems, a Braille character detection and recognition method based on Braille dot-level semantic enhancement is proposed, which includes the following:

[0006] S110, obtaining the position and category confidence of the Braille characters in the image through computer vision technology;

[0007] S120, obtaining the salient point confidence of the image pixels through computer vision technology;

[0008] S130, calculating a final confidence score using the pixel's salient point confidence score and the Braille character's coordinates and category confidence score;

[0009] S140, using a non-maximum suppression algorithm to screen the final Braille character position and category.

[0010] Furthermore, the step S110 of obtaining the position and category confidence of the Braille characters in the image by using computer vision technology specifically includes:

[0011] S1101, performing scaling preprocessing on the Braille image using bilinear interpolation;

[0012] S1102, using the RetinaNet model to perform Braille detection on the Braille image, and obtaining a coordinate set c1 = {(x1, y1, w1, h1) ... (x n ,yn , w n , h b )} and category confidence p ch ={(score 1， 1...score 1，63 )...(score n，1 ...score n，63 )};

[0013] S1103, calculate the coordinate set c2 of the Braille character in the original image according to the coordinate set c1 = (xmin1, ymin1, xmax1, ymax1) ... (xmin n , ymin n , xmax n , ymax n ).

[0014] Furthermore, the step S120 of obtaining the salient point confidence of the image pixels by using computer vision technology specifically includes:

[0015] S1201, scaling the outputs f = {f1, f2, f3, f4} of the feature pyramid method in the RetinaNet model to the same size using nearest neighbor interpolation and sequentially concatenating them to obtain f′ = (f1, f2′, f3′, f4′);

[0016] S1202, use the channel attention enhanced convolutional neural network to obtain the salient point confidence p of each pixel in the original image with a height of h and a width of w. dot ={p1, ...p h*w}.

[0017] Furthermore, step S130 calculates the final confidence using the pixel's salient point confidence and the Braille character's coordinates and category confidence, specifically including:

[0018] S1301, from p dot Constructing the integral graph 0≤x≤h, 0≤y≤w;

[0019] S1302, calculate each coordinate in c2 c2(i) = (xmin i , ymin i , xmax i , ymax i ) contains a set of 6 Braille dot coordinates, c dot (i) = {(xmin i , ymin i , xmin i +w / 2,ymin i +h / 3), (xmini , ymin i , xmin i +w / 2,ymin i +2h / 3), (xmin i , ymin i , xmin i +w / 2,ymax i ), (xmin i +w / 2,ymin i , xmax i , ymin i +h / 3),(xmin i +w / 2,ymin i , xmax i , ymin i +2h / 3), (xmin i +w / 2,ymin i , xmax i , ymax i )}, w = xmax i -xmin i +1, h=ymax i -ymin i +1;

[0020] S1303 calculates the confidence level P of each Braille point based on the integral image and the Braille point coordinates dot (i) = I(xmax i , ymax i )-I(xmax i , ymin i -1)-I(xmin i -1,ymax i )+I(xmin i -1,ymin i -1);

[0021] S1304 calculates the final confidence of the Braille character. Calculate P′ for each coordinate c2(i, 1...6) of c2(i) dot (i, 1...6)′ and perform cumulative multiplication,

[0022] Furthermore, the use of the non-maximum suppression algorithm to screen the final Braille character position and category in step S140 specifically includes:

[0023] S1401, setting the confidence threshold t=0.3;

[0024] S1402, setting the IoU threshold t of the Braille character coordinates iou=0.02;

[0025] S1403, for each Braille character in c, select the category with the highest confidence as the category of the Braille character, and delete characters with confidence less than a threshold t;

[0026] S1404, sorting all Braille characters by confidence level, and selecting the Braille character with the highest confidence level;

[0027] S1405, record the obtained Braille character as the main character. Calculate the IoU between this character and the remaining characters, delete the secondary box; delete the IoU>t iou All characters of;

[0028] S1406, return to S1404 until all characters are processed;

[0029] S1407, taking all main characters as detection results.

[0030] The second aspect of the present invention relates to a Braille character detection and recognition system based on Braille dot-level semantic enhancement, comprising:

[0031] Braille character position and category confidence acquisition module, which uses computer vision technology to obtain the position and category confidence of Braille characters in the image;

[0032] The convex point confidence acquisition module obtains the convex point confidence of image pixels through computer vision technology;

[0033] The final confidence calculation module calculates the final confidence using the pixel's salient point confidence and the Braille character's coordinates and category confidence;

[0034] The Braille character position and category screening module uses the non-maximum suppression algorithm to screen the final Braille character position and category.

[0035] The third aspect of the present invention relates to a Braille character detection and recognition device based on Braille dot-level semantic enhancement, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a Braille character detection and recognition method based on Braille dot-level semantic enhancement of the present invention.

[0036] A fourth aspect of the present invention relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method of the present invention for detecting and recognizing Braille characters based on Braille dot-level semantic enhancement is implemented.

[0037] The present invention has the beneficial effect of providing a Braille character detection and recognition method based on Braille dot-level semantic enhancement, wherein the method uses computer vision technology to obtain the position and category confidence of Braille characters in an image; uses computer vision technology to obtain the salient point confidence of image pixels; calculates the final confidence using the salient point confidence of the pixels and the coordinates and category confidence of the Braille characters; and uses a non-maximum suppression algorithm to screen the final Braille character position and category. This improves the problem in the prior art of using target detection models to detect and recognize Braille characters, which is unable to accurately judge ambiguous Braille dots. Braille dot-level semantic enhancement improves the model's ability to judge ambiguous Braille, thereby improving the accuracy of Braille character detection and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention will be further described below with reference to the accompanying drawings and examples.

[0039] Figure 1 This is a flowchart of a Braille character detection and recognition method based on Braille dot-level semantic enhancement provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0041] Example 1

[0042] like Figure 1 As shown, this embodiment 1 provides a Braille character detection and recognition method based on Braille dot-level semantic enhancement, which includes: proposing a solution to the problem of difficulty in detecting and recognizing fuzzy Braille.

[0043] Specifically, the method includes:

[0044] S110: Obtain the coordinate position and category confidence of the Braille characters in the image through computer vision technology.

[0045] Specifically, the RetinaNet model predicts the positions of the four corner points of three Braille characters for each pixel in the image, and the RetinaNet model predicts the category confidence of the three Braille characters for each pixel in the image.

[0046] S120: Using computer vision technology to obtain the salient point confidence of the image pixels.

[0047] Specifically, the output of the feature pyramid method in the RetinaNet model is used as input, the input is scaled to the same size using nearest neighbor interpolation, and then concatenated. A convolutional neural network enhanced with channel attention is then used to obtain the salient confidence of each pixel.

[0048] S130: Calculate a final confidence using the pixel's salient point confidence and the Braille character's coordinates and category confidence.

[0049] The method for calculating the final confidence using the salient point confidence of the pixel and the coordinates and category confidence of the Braille character includes:

[0050] Construct an integral image I from the pixel's salient confidence;

[0051] Calculate the set of 6 Braille dot coordinates contained in the predicted Braille character;

[0052] Calculate the confidence of each Braille dot based on the integral image and the Braille dot coordinates;

[0053] The final confidence of the Braille character is calculated by multiplying the 6 Braille point confidences of each predicted Braille character with the category confidence of the Braille character.

[0054] S140: Using a non-maximum suppression algorithm to screen the final Braille character position and category.

[0055] Specifically, the final Braille character positions and categories screened using the non-maximum suppression algorithm include:

[0056] S1401, setting the confidence threshold t=0.3;

[0057] S1402, setting the IoU threshold t of the Braille character coordinates iou =0.02;

[0058] S1403: For each Braille character in c, select the category with the highest confidence as the Braille character category, and delete characters with confidence less than a threshold t;

[0059] S1404, sorting all Braille characters by confidence level, and selecting the Braille character with the highest confidence level;

[0060] S1405, record the obtained Braille character as the main character. Calculate the IoU between this character and the remaining characters, and delete the secondary box. Delete IoU>t iou All characters of;

[0061] S1406, return to S1404, until all characters are processed;

[0062] S1407, taking all main characters as detection results;

[0063] In summary, the present invention provides a Braille character detection and recognition method based on Braille dot-level semantic enhancement. The method uses computer vision technology to obtain the location and category confidence of Braille characters in an image; also uses computer vision technology to obtain the salient point confidence of image pixels; and finally selects Braille characters based on the category confidence of the Braille character and the salient point confidence within a region. This method improves the existing problem of the inability to accurately determine ambiguous Braille dots when detecting and recognizing Braille characters using target detection models, thereby improving detection and recognition accuracy.

[0064] Example 2

[0065] This embodiment relates to a Braille character detection and recognition system based on Braille dot-level semantic enhancement, which is used to implement the Braille character detection and recognition method based on Braille dot-level semantic enhancement described in Example 1, including:

[0066] Braille character position and category confidence acquisition module, which uses computer vision technology to obtain the position and category confidence of Braille characters in the image;

[0067] The convex point confidence acquisition module obtains the convex point confidence of the image pixels through computer vision technology;

[0068] The final confidence calculation module calculates the final confidence using the pixel's salient point confidence and the Braille character's coordinates and category confidence;

[0069] The Braille character position and category screening module uses the non-maximum suppression algorithm to screen the final Braille character position and category.

[0070] Example 3

[0071] This embodiment relates to a Braille character detection and recognition device based on Braille dot-level semantic enhancement, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a Braille character detection and recognition method based on Braille dot-level semantic enhancement according to embodiment 1.

[0072] Example 4

[0073] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for detecting and recognizing Braille characters based on Braille dot-level semantic enhancement according to embodiment 1 is implemented.

[0074] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A Braille character detection and recognition method based on Braille dot-level semantic enhancement, characterized in that: The steps include: S110, obtaining the position and category confidence of the Braille characters in the image using computer vision technology; the coordinate set of the Braille characters in the original image is denoted as c2; S120, obtaining the salient point confidence of the image pixels through computer vision technology; include: S1201, scaling the outputs f = {f1, f2, f3, f4} of the feature pyramid method in the RetinaNet model to the same size using nearest neighbor interpolation and sequentially concatenating them to obtain f′ = (f1′, f2′, f3′, f4′); S1202, use the channel attention enhanced convolutional neural network to obtain the salient point confidence p of each pixel in the original image with a height of h and a width of w. dot ={p1,…p h*w }; S130, calculating a final confidence score using the pixel's salient point confidence score and the Braille character's coordinates and category confidence score; including: S1301, from p dot Constructing the integral graph S1302, calculate each coordinate in c2 c2(i) = (xmin i ,ymin i ,xmax i ,ymax i ) contains the coordinate set of 6 Braille dots, c dot (i) = {(xmin i ,ymin i ,xmin i +w / 2,ymin i +h / 3),(xmin i ,ymin i ,xmin i +w / 2,ymin i +2h / 3),(xmin i ,ymin i ,xmin i +w / 2,ymax i ),(xmin i +w / 2,ymin i ,xmax i ,ymin i +h / 3),(xmin i +w / 2,ymin i ,xmax i ,ymin i +2h / 3),(xmin i +w / 2,ymin i ,xmax i ,ymax i )},w=xmax i -xmin i +1,h=ymax i -ymin i +1; S1303, calculate the confidence level P′ of each Braille dot based on the integral image and the Braille dot coordinates dot (i) = I(xmax i ,ymax i )-I(xmax i ,ymin i -1)-I(xmin i -1,ymax i )+I(xmin i -1,ymin i -1); S1304, calculate the final confidence of the Braille character; calculate P for each coordinate c2(i,j) of c2(i) d ' ot (i,j) and perform cumulative multiplication, S140, using a non-maximum suppression algorithm to screen the final Braille character position and category; including: S1401, setting the confidence threshold t=0.3; S1402, setting the IoU threshold t of the Braille character coordinates iou =0.02; S1403, for each Braille character in c, select the category with the highest confidence as the Braille character category, and delete characters with confidence less than a threshold t; S1404, sorting all Braille characters by confidence level, and selecting the Braille character with the highest confidence level; S1405, record the obtained Braille character as the main character; calculate the IoU between this character and the remaining characters, delete the secondary box; delete the IoU>t iou All characters of; S1406, return to S1404, until all characters are processed; S1407, taking all main characters as detection results.

2. The Braille character detection and recognition method based on Braille dot-level semantic enhancement according to claim 1, characterized in that: Step S110 includes: S1101, performing scaling preprocessing on the Braille image using bilinear interpolation; S1102, using the RetinaNet model to perform Braille detection on the Braille image, and obtaining a coordinate set c1 = {(x1, y1, w1, h1) ... (x n ,y n ,w n ,h n )} and category confidence p ch ={(score 1,1 …score 1,63 )…(score n,1 …score n,63 )}; S1103, calculate the coordinate set c2 of the Braille character in the original image based on the coordinate set c1 = (xmin1, ymin1, xmax1, ymax1) ... (xmin n ,ymin n ,xmax n ,ymax n ).

3. A Braille character detection and recognition system based on Braille dot-level semantic enhancement, characterized in that: include: Braille character position and category confidence acquisition module, which uses computer vision technology to obtain the position and category confidence of Braille characters in the image; The convex point confidence acquisition module obtains the convex point confidence of the image pixels through computer vision technology; include: S1201, scaling the outputs f = {f1, f2, f3, f4} of the feature pyramid method in the RetinaNet model to the same size using nearest neighbor interpolation and sequentially concatenating them to obtain f′ = (f1′, f2′, f3′, f4′); S1202, use the channel attention enhanced convolutional neural network to obtain the salient point confidence p of each pixel in the original image with a height of h and a width of w. dot ={p1,…p h*w }; The final confidence calculation module uses the pixel convex point confidence and the coordinates and category confidence of the Braille character to calculate the final confidence; including: S1301, from p dot Constructing the integral graph S1302, calculate each coordinate in c2 c2(i) = (xmin i ,ymin i ,xmax i ,ymax i ) contains the coordinate set of 6 Braille dots, c dot (i) = {(xmin i ,ymin i ,xmin i +w / 2,ymin i +h / 3),(xmin i ,ymin i ,xmin i +w / 2,ymin i +2h / 3),(xmin i ,ymin i ,xmin i +w / 2,ymax i ),(xmin i +w / 2,ymin i ,xmax i ,ymin i +h / 3),(xmin i +w / 2,ymin i ,xmax i ,ymin i +2h / 3),(xmin i +w / 2,ymin i ,xmax i ,ymax i )},w=xmax i -xmin i +1,h=ymax i -ymin i +1; where c2 is the coordinate set of the Braille character in the original image; S1303, calculate the confidence level P′ of each Braille dot based on the integral image and the Braille dot coordinates dot (i) = I(xmax i ,ymax i )-I(xmax i ,ymin i -1)-I(xmin i -1,ymax i )+I(xmin i -1,ymin i -1); S1304, calculate the final confidence of the Braille character; calculate P′ for each coordinate c2(i,j) of c2(i) dot (i,j) and perform cumulative multiplication, The Braille character position and category screening module uses the non-maximum suppression algorithm to screen the final Braille character position and category; including: S1401, setting the confidence threshold t=0.3; S1402, setting the IoU threshold t of the Braille character coordinates iou =0.02; S1403, for each Braille character in c, select the category with the highest confidence as the Braille character category, and delete characters with confidence less than a threshold t; S1404, sorting all Braille characters by confidence level, and selecting the Braille character with the highest confidence level; S1405, record the obtained Braille character as the main character; calculate the IoU between this character and the remaining characters, delete the secondary box; delete the IoU>t iou All characters of; S1406, return to S1404, until all characters are processed; S1407, taking all main characters as detection results.

4. A Braille character detection and recognition device based on Braille dot-level semantic enhancement, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a Braille character detection and recognition method based on Braille dot-level semantic enhancement as described in any one of claims 1-2.

5. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, a Braille character detection and recognition method based on Braille dot-level semantic enhancement according to any one of claims 1-2 is implemented.

Images