Blind point graph automatic conversion method and device based on multi-modal large model and optical character recognition

Abstract

The application discloses a blind point graph automatic conversion method and device based on a multimodal large model and optical character recognition, and the method comprises the following steps: collecting a picture-text mixed layout image of a scene and labeling text and graphic information, and constructing a picture-text mixed layout special data set; using a model to extract features from a preprocessed image and detect a text region, and outputting a polygon coordinate frame to realize accurate positioning of a text position of the image; cutting a corresponding region according to a text frame coordinate, using a CNN to extract text image features, modeling a character context relationship through an RNN, and then decoding through a CTC decoder to complete recognition and extraction of the image text; receiving the recognized text, first cleaning and word segmentation preprocessing, then modeling semantics through a Transformer model, and gradually generating a Braille character sequence through a decoder to realize translation and conversion of the text into Braille; integrating original image, mask and Braille and the like to design a special prompt, inputting the big model, freezing and fine-tuning model parameters through LoRA technology, and converting the original picture content into a standard Braille point graph.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and information accessibility technology, specifically a method and device for automatic conversion of Braille dot images in teaching scenarios that combines multimodal large models and optical character recognition (OCR) technology. It is applicable to the conversion and graphic processing of mixed text and image images in teaching scenarios, and involves technical branches such as artificial intelligence model application, image recognition and generation algorithms, and Braille information processing. Background Technology

[0002] In the field of special education, visually impaired students rely on Braille textbooks for learning, but traditional Braille textbooks have significant shortcomings in handling content that combines text and graphics. Currently, the presentation of graphic information in Braille textbooks is limited, making it difficult to effectively convert two-dimensional graphics into Braille dotted information. This results in visually impaired students facing enormous difficulties in learning subjects such as mathematics and physics, which rely on graphic comprehension.

[0003] In existing technologies, some methods attempt to automate the processing of dot maps using image recognition technology, but these methods have several shortcomings. Traditional OCR technology suffers from low accuracy in complex graphic scenarios and cannot effectively understand the overall semantics of sentences. Furthermore, when multimodal large models are directly applied to the conversion of dot maps for the visually impaired, accurate conversion is difficult due to the limited availability of specialized data on dot maps and the lack of optimization tailored to their structural characteristics and semantic requirements. In addition, in scenarios involving mixed text and images, the interference between geometric shapes and text makes layout analysis difficult, and text obscuring the background also presents challenges in restoring the graphic structure. Existing technologies are insufficient to meet the needs of accessible information processing for visually impaired students in educational settings.

[0004] The above-mentioned problems urgently need to be solved. Summary of the Invention

[0005] To overcome the above-mentioned shortcomings of the prior art, this invention provides a method and apparatus for automatic conversion of blind dot maps based on the advantages of multimodal large model image and text understanding.

[0006] To address the aforementioned technical challenges, a blind dot map automatic conversion method based on multimodal large model and optical character recognition is proposed, comprising the following steps:

[0007] S110: Collect images of mixed text and graphics in the scene and label them with text and graphic information to build a dedicated dataset for mixed text and graphics.

[0008] S120 uses the model to extract features from the preprocessed image and detect text regions, outputting polygonal coordinate boxes to achieve precise positioning of text in the image;

[0009] S130: The corresponding area is cropped according to the coordinates of the text box, the text image features are extracted using CNN, the character context relationship is modeled by RNN, and then the CTC decoder is used to decode the image text to complete the recognition and extraction of the text.

[0010] S140 receives and recognizes text, first cleans and preprocesses it by word segmentation, then models the semantics through the Transformer model, and gradually generates Braille character sequences by the decoder to realize the translation and conversion of text to Braille;

[0011] S150 integrates information such as the original image, mask, and Braille to design a dedicated prompt. Input a large model, freeze and fine-tune the model parameters through LoRA technology, and convert the original image content into a Braille dot map that conforms to the specifications.

[0012] Furthermore, step S110, which involves constructing a dedicated dataset for mixed text and image layout and preprocessing, specifically includes:

[0013] S1101, collect mixed images of text and graphics containing geometric shapes and physical diagrams in the scene, covering scenes such as scanned copies of teaching materials and screenshots of blackboard writing, to form the initial dataset D;

[0014] S1102, perform pixel-level annotation on text regions and geometric vertices / edges in the image, generating an annotation set L = {L text ,L graph}

[0015] Furthermore, the image text detection described in step S120 specifically includes:

[0016] S1201, normalize the input image. in

[0017] I(x,y) represents the pixel value of the original image, and μ and σ represent the mean and variance of the entire training image set.

[0018] S1202, extracting multi-scale feature maps from normalized images based on MobileNet;

[0019] S1203, the multi-scale feature map generates a probability map and a threshold map through a differentiable binarization detection head (DB). The probability map represents the probability that each pixel belongs to a text region, and the threshold map dynamically assigns a binarization threshold to each location;

[0020] S1204 extracts the contours of the binary image, generates polygon vertices, and outputs a polygonal bounding box composed of multiple coordinate points.

[0021] Furthermore, the image text recognition and extraction described in step S130 specifically includes:

[0022] S1301, parse the coordinates of the text box generated by S120, and crop out the text image area;

[0023] S1302, CNN extracts text image representations, and RNN models the feature sequences extracted by CNN;

[0024] S1303, the CTC decoder decodes the character sequence and outputs the final text;

[0025] Furthermore, step S140, translating the extracted text into Braille, specifically includes:

[0026] S1401: Receive the recognized text content output by S130 and perform preprocessing operations such as text cleaning and word segmentation on the text;

[0027] S1402, Load the pre-trained Transformer model to perform semantic modeling on the input text;

[0028] S1403 uses a Transformer decoder to generate a sequence of Braille characters step by step based on the encoded information.

[0029] Furthermore, the graphics transformation and generation based on the multimodal large model described in step S150 specifically includes:

[0030] S1501, Receive the original image, mask image, translated Braille, design a special prompt, and composition condition information from the previous steps;

[0031] S1502 uses a Stable Diffusion large model. The input is the original image and conditional information, and the output is the replaced / generated image, i.e., blind point map.

[0032] S1503 uses LoRA technology to freeze parameters and fine-tune large models;

[0033] S1504 uses a finely tuned large model to convert the Chinese characters in the original image into a dotted image for the blind.

[0034] A second aspect of the present invention relates to an automatic conversion device for blind dot maps based on multimodal large model and optical character recognition, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the automatic conversion method for blind dot maps based on multimodal large model and optical character recognition of the present invention.

[0035] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the blind dot map automatic conversion method based on multimodal large model and optical character recognition of the present invention.

[0036] This invention achieves accurate localization of text regions in images by introducing a text detection module (S120), completes high-precision text recognition by employing a text recognition module (S130) based on CRNN and CTC loss function, and further converts the recognition results into Braille representation through a Braille translation module (S140), thereby achieving barrier-free information conversion. In terms of image generation, this invention utilizes a Stable Diffusion model (S150) to fuse image and text understanding to perform text dot map replacement on the original image, ultimately completing the automatic conversion of Braille dot maps.

[0037] The innovation of this invention lies in integrating multimodal large models with deep learning OCR technology to construct a complete process system of "text detection - text recognition - Braille translation - graphic filling", which solves the problems of low recognition rate, inaccurate Braille conversion, and difficulty in graphic restoration in traditional methods in scenarios with mixed text and images.

[0038] The beneficial effects of this invention are that it effectively improves the accuracy of text recognition and the restoration of graphic structure in mixed text and image scenarios, overcoming the problems of low recognition rate and difficult graphic processing in traditional technologies in mixed text and image scenarios; at the same time, it realizes accurate semantic conversion from text to Braille, and generates standardized Braille dot maps by combining large models, effectively meeting the needs of visually impaired groups for barrier-free access to text and image information. Attached Figure Description

[0039] Figure 1 This is a flowchart of the automatic conversion method for blind dot maps in teaching scenarios based on multimodal large model and optical character recognition technology provided in the embodiments of the present invention.

[0040] Figure 2 This is a schematic diagram of the device of the present invention. Detailed Implementation

[0041] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0042] Example 1

[0043] like Figure 1 As shown in the figure, this embodiment 1 provides a method for automatic conversion of blind dot graphs based on multimodal large model and optical character recognition for use in teaching scenarios. The method improves the problem that the existing technology has a low text recognition rate in complex text and image mixed backgrounds and that text occlusion of the background makes it difficult to restore the graphic structure.

[0044] Specifically, the method includes:

[0045] S110: Collect images of mixed text and graphics in teaching scenarios and label them with text and graphic information to build a dedicated dataset for mixed text and graphics.

[0046] Specifically, a large number of images containing geometric figures and physics diagrams in teaching scenarios were collected. These images were obtained from various scenarios, including scanned copies of textbooks and screenshots of blackboard writing, to ensure coverage of as many real-world usage scenarios as possible. Then, these images underwent meticulous annotation, including pixel-level annotations of text regions and the vertices / edges of geometric figures.

[0047] S120 uses the model to extract features from the preprocessed image and detect text regions, outputting polygonal coordinate boxes to achieve precise positioning of text in the image;

[0048] Specifically, after normalizing the input image, MobileNet is used as the backbone network to extract multi-scale feature maps. This network structure is lightweight and has good feature representation capabilities. Then, a differentiable binarization detection head (DB) is used to generate probability maps and threshold maps. The probability map represents the likelihood of each pixel belonging to a text region, while the threshold map dynamically assigns an appropriate binarization threshold to each location. The process can be simply summarized as follows:

[0049] P=σ(F prob (x)),T=σ(F thresh (x))

[0050] Where σ is the sigmoid function, mapping the output between (0, 1), F prob and F thresh These are the neural network modules that generate the probability map and the threshold map, respectively.

[0051] Finally, contour extraction is performed on the obtained binary image to generate a polygonal box composed of multiple coordinate points for accurately locating the text in the image.

[0052] S130: The corresponding area is cropped according to the coordinates of the text box, the text image features are extracted using CNN, the character context relationship is modeled by RNN, and then the CTC decoder is used to decode the image text to complete the recognition and extraction of the text.

[0053] Specifically, based on the text box coordinates output in the previous step, the corresponding text image region is cropped, and its high-dimensional visual features are extracted using a CNN network. These feature sequences are then fed into an RNN for modeling to capture the contextual dependencies between characters. Finally, an end-to-end character sequence decoding is achieved through a CTC decoder, outputting the recognition result and completing the accurate extraction of text content from the image.

[0054] S140 receives and recognizes text, first cleans and preprocesses it by word segmentation, then models the semantics through the Transformer model, and gradually generates Braille character sequences by the decoder to realize the translation and conversion of text to Braille;

[0055] Specifically, first, preprocessing operations such as cleaning and tokenization are performed on the text content output by S130 to remove irrelevant characters and possible recognition errors. Then, a pre-trained Transformer model is loaded to perform semantic modeling on the input text, making full use of its powerful context understanding ability. The Transformer is divided into an encoder and a decoder.

[0056] The encoder outputs the representation H = Encoder(x) = LayerNorm(x + FFN(MultiHeadAttn(x, x, x))). The decoder predicts the Braille character distribution p(y t ∣y <t , x) = Softmax(DecoderLayer(y <t , H))

[0057] Finally, the Transformer decoder is used to gradually generate a character sequence that conforms to the Braille specification based on the encoded information.

[0058] S150, integrate information such as the original image, mask, and Braille to design a dedicated prompt, input it into the large model, freeze and fine-tune the model parameters through LoRA technology, and convert the content of the original image into a Braille dot map that conforms to the specification;

[0059] Specifically, receive the original image, mask image, and translated Braille from the previous steps, etc., and design a dedicated prompt to form conditional information. Feed these conditional information as input into the Stable Diffusion large model to guide it to generate a Braille dot map that meets the requirements. To improve the performance of the model on specific tasks, we use LoRA technology to freeze some parameters and perform fine-tuning to make it better adapt to the requirements of Braille dot map generation. The basic idea of LoRA is to add a low-rank matrix ΔW to the original weight matrix W, such that: W' = W + ΔW = W + AB T where A ∈ R d×r , B ∈ R k×r , and r << d, k, so that the model behavior can be adjusted without significantly increasing the number of parameters, making it more suitable for the Braille dot map generation task.

[0060] Finally, use the fine-tuned large model to convert the Chinese character content in the original image into a Braille dot map while retaining the overall structure of the image.

[0061] In summary, this embodiment provides an automatic conversion method for Braille dot maps in teaching scenarios based on multimodal large-scale models and optical character recognition technology. This method combines a text detection module based on MobileNet and a differentiable binarization detection head to achieve high-precision localization of text regions in teaching images. Furthermore, it employs a CNN-RNN-CTC architecture to achieve robust text recognition in complex scenarios, and translates the recognition results into character sequences conforming to Braille standards using a Transformer model. Finally, it introduces a Stable Diffusion multimodal generation model, combined with a LoRA parameter fine-tuning strategy, to fuse mask information and Braille semantics to generate Braille dot maps based on the original image. The entire process effectively solves the problems of low recognition rate, difficulty in restoring graphic structures, and inaccurate Braille conversion in existing technologies when processing mixed text and image images in teaching scenarios, demonstrating good practicality and promotional value.

[0062] Example 2

[0063] like Figure 2 This embodiment provides an automatic conversion device for blind dot maps based on multimodal large model and optical character recognition, including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the automatic conversion method for blind dot maps based on multimodal large model and optical character recognition of Embodiment 1.

[0064] Example 3

[0065] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the blind dot map automatic conversion method based on multimodal large model and optical character recognition as described in Embodiment 1.

[0066] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. An automatic dot map conversion method for blind users based on multimodal large model and optical character recognition, comprising the following steps: S110: Collect images of mixed text and graphics in the scene and label them with text and graphic information to build a dedicated dataset for mixed text and graphics. S120 uses the model to extract features from the preprocessed image and detect text regions, outputting polygonal coordinate boxes to achieve precise positioning of text in the image; S130: The corresponding area is cropped according to the coordinates of the text box, the text image features are extracted using CNN, the character context relationship is modeled by RNN, and then decoded by CTC decoder to complete the recognition and extraction of image text; S140 receives and recognizes text, first performs text cleaning and word segmentation preprocessing, then models semantics using a Transformer model, and the decoder gradually generates Braille character sequences to achieve text-to-Braille translation; specifically including: S1401: Receive the recognized text content output by S130, and perform text cleaning and word segmentation preprocessing operations. S1402, Load the pre-trained Transformer model to perform semantic modeling on the input text; S1403 uses a Transformer decoder to generate a sequence of Braille characters step by step based on the encoded information; S150 integrates information from the original image, mask, and Braille character sequence to design a dedicated prompt. It takes a large input model, freezes and fine-tunes model parameters using LoRA technology, and converts the original image content into a standardized Braille dot map. Specifically, it includes: S1501: Receive the original image, mask, and Braille character sequence from the previous steps, design a dedicated prompt, and compose condition information. S1502 uses a Stable Diffusion large model. The input is the original image and conditional information, and the output is the replaced / generated image, i.e., blind point map. S1503 uses LoRA technology to freeze parameters and fine-tune large models; S1504 uses a finely tuned large model to convert the Chinese characters in the original image into a dotted image for the blind.

2. The method for automatic conversion of blind dot maps based on multimodal large model and optical character recognition as described in claim 1, characterized in that, Step S110, which involves constructing a dedicated dataset for mixed text and image layout, specifically includes: S1101, collect mixed images of text and graphics containing geometric shapes and physical diagrams in the scene, covering the scene of scanned textbooks and screenshots of blackboard writing, to form the initial dataset D; S1102, perform pixel-level annotation on text regions and geometric vertices / edges in the image, generating an annotation set L={L text ,L graph } 3. The method for automatic conversion of blind dot maps based on multimodal large model and optical character recognition as described in claim 1, characterized in that, The image text detection in step S120 specifically includes: S1201, Normalize the input image; Where I(x,y) represents the pixel values ​​of the original image. , Represents the mean and variance of the entire training image set; S1202, extracting multi-scale feature maps from normalized images based on MobileNet; S1203, the multi-scale feature map generates a probability map and a threshold map through the differentiable binarization detection head DB; the probability map represents the probability that each pixel belongs to the text region, and the threshold map dynamically assigns a binarization threshold to each position; S1204 extracts the contours of the binary image, generates polygon vertices, and outputs a polygon coordinate frame composed of multiple coordinate points.

4. The method for automatic conversion of blind dot maps based on multimodal large model and optical character recognition as described in claim 1, characterized in that, The image text recognition and extraction described in step S130 specifically includes: S1301, parse the polygonal coordinate frame generated by S120, and crop out the text image region; S1302, CNN extracts text image representations, and RNN models the feature sequences extracted by CNN; S1303, the CTC decoder decodes the character sequence and outputs the final text.

5. A blind dot map automatic conversion device based on multimodal large model and optical character recognition, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the blind dot map automatic conversion method based on multimodal large model and optical character recognition as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the blind dot map automatic conversion method based on multimodal large model and optical character recognition as described in any one of claims 1-4.

Images