Blind dot diagram automatic conversion method and device based on multi-mode large model and optical character recognition

Abstract

The invention discloses a blind point diagram automatic conversion method and device based on a multi-mode large model and optical character recognition, and the method comprises the steps: collecting image-text mixed arrangement images of a scene, marking text and graphic information, and constructing a special image-text mixed arrangement data set; extracting features of the preprocessed image by using a model, detecting a text area, outputting a polygonal coordinate frame, and realizing accurate positioning of a text position in the image; cutting a corresponding area according to a textbox coordinate, extracting a text image feature by using a CNN, modeling a character context relationship through an RNN, and decoding by a CTC decoder to complete identification and extraction of an image text; the method comprises the following steps: receiving a recognition text, firstly performing cleaning and word segmentation preprocessing, then modeling semantics through a Transform model, gradually generating a braille character sequence by a decoder, and realizing translation conversion from the text to the braille; a special prompt is designed by integrating information such as an original image, masks and braille alphabets, a large model is input, model parameters are frozen and fine-tuned through the LoRA technology, and the content of the original image is converted into a blind point diagram conforming to specifications.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and information accessibility technology, specifically a method and device for automatic conversion of Braille dot images in teaching scenarios that combines multimodal large models and optical character recognition (OCR) technology. It is applicable to the conversion and graphic processing of mixed text and image images in teaching scenarios, and involves technical branches such as artificial intelligence model application, image recognition and generation algorithms, and Braille information processing. Background Technology

[0002] In the field of special education, visually impaired students rely on Braille textbooks for learning, but traditional Braille textbooks have significant shortcomings in handling content that combines text and graphics. Currently, the presentation of graphic information in Braille textbooks is limited, making it difficult to effectively convert two-dimensional graphics into Braille dotted information. This results in visually impaired students facing enormous difficulties in learning subjects such as mathematics and physics, which rely on graphic comprehension.

[0003] In existing technologies, some methods attempt to automate the processing of dot maps using image recognition technology, but these methods have many shortcomings. Traditional OCR technology has low recognition accuracy in complex graphic scenarios and cannot effectively understand the overall semantics of sentences. When multimodal large models are directly applied to the conversion of dot maps for the visually impaired, accurate conversion is difficult to achieve due to the limited availability of professional data on dot maps for the visually impaired and the lack of optimization for their structural characteristics and semantic requirements. Furthermore, in scenarios with mixed text and images, the interference between geometric shapes and text makes layout analysis difficult, and the text obscuring the background also poses challenges to the reconstruction of graphic structures. Existing technologies are insufficient to meet the needs of accessible information processing for visually impaired students in educational settings.

[0004] The above-mentioned problems urgently need to be solved. Summary of the Invention

[0005] To overcome the above-mentioned shortcomings of the prior art, this invention provides a method and apparatus for automatic conversion of blind dot maps based on the advantages of multimodal large model image and text understanding.

[0006] To address the aforementioned technical challenges, a blind dot map automatic conversion method based on multimodal large model and optical character recognition is proposed, comprising the following steps:

[0007] S110: Collect images of mixed text and graphics in the scene and label them with text and graphic information to build a dedicated dataset for mixed text and graphics.

[0008] S120 uses the model to extract features from the preprocessed image and detect text regions, outputting polygonal coordinate boxes to achieve precise positioning of text in the image;

[0009] S130: The corresponding area is cropped according to the coordinates of the text box, the text image features are extracted using CNN, the character context relationship is modeled by RNN, and then the CTC decoder is used to decode the image text to complete the recognition and extraction of the text.

[0010] S140 receives and recognizes text, first cleans and preprocesses it by word segmentation, then models the semantics through the Transformer model, and gradually generates Braille character sequences by the decoder to realize the translation and conversion of text to Braille;

[0011] S150 integrates information such as the original image, mask, and Braille to design a dedicated prompt. Input a large model, freeze and fine-tune the model parameters through LoRA technology, and convert the original image content into a Braille dot map that conforms to the specifications.

[0012] Furthermore, step S110, which involves constructing a dedicated dataset for mixed text and image layout and preprocessing, specifically includes:

[0013] S1101, collect mixed images of text and graphics containing geometric shapes and physical diagrams in the scene, covering scenes such as scanned copies of teaching materials and screenshots of blackboard writing, to form the initial dataset D;

[0014] S1102, perform pixel-level annotation on text regions and geometric vertices / edges in the image, generating an annotation set L = {L text ,L graph}

[0015] Furthermore, the image text detection described in step S120 specifically includes:

[0016] S1201, normalize the input image. in

[0017] I(x,y) represents the pixel value of the original image, and μ and σ represent the mean and variance of the entire training image set.

[0018] S1202, extracting multi-scale feature maps from normalized images based on MobileNet;

[0019] S1203, the multi-scale feature map generates a probability map and a threshold map through a differentiable binarization detection head (DB). The probability map represents the probability that each pixel belongs to a text region, and the threshold map dynamically assigns a binarization threshold to each location;

[0020] S1204 extracts the contours of the binary image, generates polygon vertices, and outputs a polygonal bounding box composed of multiple coordinate points.

[0021] Furthermore, the image text recognition and extraction described in step S130 specifically includes:

[0022] S1301, parse the coordinates of the text box generated by S120, and crop out the text image area;

[0023] S1302, CNN extracts text image representations, and RNN models the feature sequences extracted by CNN;

[0024] S1303, the CTC decoder decodes the character sequence and outputs the final text;

[0025] Furthermore, step S140, translating the extracted text into Braille, specifically includes:

[0026] S1401: Receive the recognized text content output by S130, and perform preprocessing operations such as text cleaning and word segmentation on the text;

[0027] S1402, Load the pre-trained Transformer model to perform semantic modeling on the input text;

[0028] S1403 uses a Transformer decoder to generate a sequence of Braille characters step by step based on the encoded information.

[0029] Furthermore, the graphics transformation and generation based on the multimodal large model described in step S150 specifically includes:

[0030] S1501, Receive the original image, mask image, translated Braille, design a special prompt, and composition condition information from the previous steps;

[0031] S1502 uses a Stable Diffusion large model. The input is the original image and conditional information, and the output is the replaced / generated image, i.e., blind point map.

[0032] S1503 uses LoRA technology to freeze parameters and fine-tune large models;

[0033] S1504 uses a finely tuned large model to convert the Chinese characters in the original image into a dotted image for the blind.

[0034] A second aspect of the present invention relates to an automatic conversion device for blind dot maps based on multimodal large model and optical character recognition, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the automatic conversion method for blind dot maps based on multimodal large model and optical character recognition of the present invention.

[0035] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the blind dot map automatic conversion method based on multimodal large model and optical character recognition of the present invention.

[0036] This invention achieves accurate localization of text regions in images by introducing a text detection module (S120), completes high-precision text recognition by employing a text recognition module (S130) based on CRNN and CTC loss function, and further converts the recognition results into Braille representation through a Braille translation module (S140), thereby achieving barrier-free information conversion. In terms of image generation, this invention utilizes a Stable Diffusion model (S150) to fuse image and text understanding to perform text dot map replacement on the original image, ultimately completing the automatic conversion of Braille dot maps.

[0037] The innovation of this invention lies in integrating multimodal large models with deep learning OCR technology to construct a complete process system of "text detection - text recognition - Braille translation - graphic filling", which solves the problems of low recognition rate, inaccurate Braille conversion, and difficulty in graphic restoration in traditional methods in scenarios with mixed text and images.

[0038] The beneficial effects of this invention are that it effectively improves the accuracy of text recognition and the restoration of graphic structure in mixed text and image scenarios, overcoming the problems of low recognition rate and difficult graphic processing in traditional technologies in mixed text and image scenarios; at the same time, it realizes accurate semantic conversion from text to Braille, and generates standardized Braille dot maps by combining large models, effectively meeting the needs of visually impaired groups for barrier-free access to text and image information. Attached Figure Description

[0039] Figure 1 This is a flowchart of the automatic conversion method for blind dot maps in teaching scenarios based on multimodal large model and optical character recognition technology provided in the embodiments of the present invention.

[0040] Figure 2 This is a schematic diagram of the device of the present invention. Detailed Implementation

[0041] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0042] Example 1

[0043] like Figure 1 As shown in the figure, this embodiment 1 provides a method for automatic conversion of blind dot graphs based on multimodal large model and optical character recognition for use in teaching scenarios. The method improves the problem that the existing technology has a low text recognition rate in complex text and image mixed backgrounds and that text occlusion of the background makes it difficult to restore the graphic structure.

[0044] Specifically, the method includes:

[0045] S110: Collect images of mixed text and graphics in teaching scenarios and label them with text and graphic information to build a dedicated dataset for mixed text and graphics.

[0046] Specifically, a large number of images containing geometric figures and physics diagrams in teaching scenarios were collected. These images were obtained from various scenarios, including scanned copies of textbooks and screenshots of blackboard writing, to ensure coverage of as many real-world usage scenarios as possible. Then, these images underwent meticulous annotation, including pixel-level annotations of text regions and the vertices / edges of geometric figures.

[0047] S120 uses the model to extract features from the preprocessed image and detect text regions, outputting polygonal coordinate boxes to achieve precise positioning of text in the image;

[0048] Specifically, after normalizing the input image, MobileNet is used as the backbone network to extract multi-scale feature maps. This network structure is lightweight and has good feature representation capabilities. Then, a differentiable binarization detection head (DB) is used to generate probability maps and threshold maps. The probability map represents the likelihood of each pixel belonging to a text region, while the threshold map dynamically assigns an appropriate binarization threshold to each location. The process can be simply summarized as follows:

[0049] P=σ(F prob (x)),T=σ(F thresh (x))

[0050] Where σ is the sigmoid function, mapping the output between (0, 1), F prob and F thresh These are the neural network modules that generate the probability map and the threshold map, respectively.

[0051] Finally, contour extraction is performed on the obtained binary image to generate a polygonal box composed of multiple coordinate points for accurately locating the text in the image.

[0052] S130: The corresponding area is cropped according to the coordinates of the text box, the text image features are extracted using CNN, the character context relationship is modeled by RNN, and then the CTC decoder is used to decode the image text to complete the recognition and extraction of the text.

[0053] Specifically, based on the text box coordinates output in the previous step, the corresponding text image region is cropped, and its high-dimensional visual features are extracted using a CNN network. These feature sequences are then fed into an RNN for modeling to capture the contextual dependencies between characters. Finally, an end-to-end character sequence decoding is achieved through a CTC decoder, outputting the recognition result and completing the accurate extraction of text content from the image.

[0054] S140 receives and recognizes text, first cleans and preprocesses it by word segmentation, then models the semantics through the Transformer model, and gradually generates Braille character sequences by the decoder to realize the translation and conversion of text to Braille;

[0055] Specifically, first, preprocessing operations such as cleaning and tokenizing the text content output by S130 are performed to remove irrelevant characters and possible recognition errors. Then, a pre-trained Transformer model is loaded to perform semantic modeling on the input text, making full use of its powerful context understanding ability. The Transformer is divided into an encoder and a decoder.

[0056] The encoder outputs a representation H = Encoder(x) = LayerNorm(x + FFN(MultiHeadAttn(x, x, x))). The decoder predicts the Braille character distribution p(y t ∣y <t , x) = Softmax(DecoderLayer(y <t , H))

[0057] Finally, the Transformer decoder is used to gradually generate a character sequence that conforms to the Braille specification based on the encoded information.

[0058] In S150, information such as the original image, mask, and Braille is integrated to design a dedicated prompt, which is input into the large model. Through the LoRA technique, the model parameters are frozen and fine-tuned to convert the original image content into a Braille dot map that conforms to the specification.

[0059] Specifically, the original image, mask image, and translated Braille from the previous steps are received, and a dedicated prompt is designed to form conditional information. These conditional information are used as inputs and fed into the Stable Diffusion large model to guide it to generate a Braille dot map that meets the requirements. To improve the performance of the model on specific tasks, we use the LoRA technique to freeze some parameters and perform fine-tuning to make it better adapt to the requirements of Braille dot map generation. The basic idea of LoRA is to add a low-rank matrix ΔW to the original weight matrix W, such that: W' = W + ΔW = W + AB T where A ∈ R d×r , B ∈ R k×r , and r << d, k. In this way, the model behavior can be adjusted without significantly increasing the number of parameters, making it more suitable for the Braille dot map generation task.

[0060] Finally, the fine-tuned large model is used to convert the Chinese character content in the original image into a Braille dot map while retaining the overall structure of the image.

[0061] In summary, this embodiment provides an automatic conversion method for Braille dot maps in teaching scenarios based on multimodal large-scale models and optical character recognition technology. This method combines a text detection module based on MobileNet and a differentiable binarization detection head to achieve high-precision localization of text regions in teaching images. Furthermore, it employs a CNN-RNN-CTC architecture to achieve robust text recognition in complex scenarios, and translates the recognition results into character sequences conforming to Braille standards using a Transformer model. Finally, it introduces a Stable Diffusion multimodal generation model, combined with a LoRA parameter fine-tuning strategy, to fuse mask information and Braille semantics to generate Braille dot maps based on the original image. The entire process effectively solves the problems of low recognition rate, difficulty in restoring graphic structures, and inaccurate Braille conversion in existing technologies when processing mixed text and image images in teaching scenarios, demonstrating good practicality and promotional value.

[0062] Example 2

[0063] like Figure 2 This embodiment provides an automatic conversion device for blind dot maps based on multimodal large model and optical character recognition, including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the automatic conversion method for blind dot maps based on multimodal large model and optical character recognition of Embodiment 1.

[0064] Example 3

[0065] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the blind dot map automatic conversion method based on multimodal large model and optical character recognition as described in Embodiment 1.

[0066] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. An automatic dot map conversion method for blind users based on multimodal large model and optical character recognition, comprising the following steps: S110: Collect images of mixed text and graphics in the scene and label them with text and graphic information to build a dedicated dataset for mixed text and graphics. S120 uses the model to extract features from the preprocessed image and detect text regions, outputting polygonal coordinate boxes to achieve precise positioning of text in the image; S130: The corresponding area is cropped according to the coordinates of the text box, the text image features are extracted using CNN, the character context relationship is modeled by RNN, and then the CTC decoder is used to decode the image text to complete the recognition and extraction of the text. S140 receives and recognizes text, first cleans and preprocesses it by word segmentation, then models the semantics through the Transformer model, and gradually generates Braille character sequences by the decoder to realize the translation and conversion of text to Braille; The S150 integrates original image, mask, and Braille information into a dedicated prompt. Inputting a large model, it freezes and fine-tunes model parameters using LoRA technology, converting the original image content into a compliant Braille dot map.

2. The method for automatic conversion of blind dot maps based on multimodal large model and optical character recognition as described in claim 1, characterized in that, Step S110, which involves constructing a dedicated dataset for mixed text and image layout and preprocessing, specifically includes: S1101, collect mixed images of text and graphics containing geometric shapes and physical diagrams in the scene, covering scenes such as scanned copies of teaching materials and screenshots of blackboard writing, to form the initial dataset D; S1102, perform pixel-level annotation on text regions and geometric vertices / edges in the image, generating an annotation set L = {L text ,L graph } 3. The method for automatic conversion of blind dot maps based on multimodal large model and optical character recognition as described in claim 1, characterized in that, The image text detection described in step S120 specifically includes: S1201, Normalize the input image; Where I(x,y) represents the pixel value of the original image, and μ and σ represent the mean and variance of the entire training image set; S1202, extracting multi-scale feature maps from normalized images based on MobileNet; S1203, the multi-scale feature map generates a probability map and a threshold map through the differentiable binarization detection head DB; the probability map represents the probability that each pixel belongs to the text region, and the threshold map dynamically assigns a binarization threshold to each position; S1204 extracts the contours of the binary image, generates polygon vertices, and outputs a polygonal bounding box composed of multiple coordinate points.

4. The method for automatic conversion of blind dot maps based on multimodal large model and optical character recognition as described in claim 1, characterized in that, Step S130, image text recognition and extraction, specifically includes: S1301, parse the coordinates of the text box generated by S120, and crop out the text image area; S1302, CNN extracts text image representations, and RNN models the feature sequences extracted by CNN; S1303, the CTC decoder decodes the character sequence and outputs the final text.

5. The method for automatic conversion of blind dot maps based on multimodal large model and optical character recognition as described in claim 1, characterized in that, Step S140, translating the extracted text into Braille, specifically includes: S1401: Receive the recognized text content output by S130 and perform preprocessing operations such as text cleaning and word segmentation on the text; S1402, Load the pre-trained Transformer model to perform semantic modeling on the input text; S1403 uses a Transformer decoder to generate a sequence of Braille characters step by step based on the encoded information.

6. The method for automatic conversion of blind dot maps based on multimodal large model and optical character recognition as described in claim 1, characterized in that, Step S150, the graph transformation and generation based on a multimodal large model, specifically includes: S1501, Receive the original image, mask image, translated Braille, design a special prompt, and composition condition information from the previous steps; S1502 uses a Stable Diffusion large model. The input is the original image and conditional information, and the output is the replaced / generated image, i.e., blind point map. S1503 uses LoRA technology to freeze parameters and fine-tune large models; S1504 uses a finely tuned large model to convert the Chinese characters in the original image into a dotted image for the blind.

7. A blind dot map automatic conversion device based on multimodal large model and optical character recognition, characterized in that, The method includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the blind dot map automatic conversion method based on multimodal large model and optical character recognition as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the blind dot map automatic conversion method based on multimodal large model and optical character recognition as described in any one of claims 1-6.

Images