A method for mermaid picture analysis and understanding based on multimodal large model LoRA fine-tuning

Abstract

The application discloses a method for Mermaid picture analysis and understanding based on multimodal large model LoRA fine tuning, and belongs to the technical field of artificial intelligence, multimodal document understanding, chart analysis and code generation. The method is directed to flowchart, time sequence chart, class diagram, entity relationship diagram, user journey chart, Gantt chart, pie chart, quadrant chart, Git chart, timeline chart and XY chart and other Mermaid pictures. Through self-synthesized image-description-code triple data set construction, real picture white background completion and super-resolution enhancement, Unsloth LoRA fine tuning based on Qwen3.5-27B multimodal large model, and JSON structured output and Mermaid syntax verification post-processing, the method realizes automatic analysis from input Mermaid picture to picture category, Chinese semantic description and renderable Mermaid source code. The application can alleviate the problem that the transparent background, low resolution, small font size, arrow direction and figure relationship of real Mermaid pictures are difficult to identify, and improve the joint restoration ability of multimodal large models to chart structure, semantic relationship and code syntax.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, multimodal large models, chart parsing, document intelligence, and code generation, specifically relating to a method for parsing and understanding Mermaid chart images. This method achieves automated recovery from images to chart categories, Chinese descriptions, and Mermaid source code through synthesizing Mermaid image datasets, real image preprocessing, efficient fine-tuning of LoRA parameters based on the Qwen3.5-27B multimodal large model, and structured output verification. Background Technology

[0002] Mermaid is a text-based diagramming language widely used in software design documents, technical specifications, project management, and knowledge base systems. It can generate flowcharts, sequence diagrams, class diagrams, entity-relationship diagrams, Gantt charts, Git branch diagrams, and other diagrams using concise code. With the proliferation of Mermaid diagram images in technical documentation and collaboration platforms, the challenge of reconstructing Mermaid source code from existing images and generating understandable Chinese descriptions has become a crucial issue in document automation, knowledge base migration, and diagram editability.

[0003] Existing technologies typically employ OCR recognition, image segmentation, rule-based connection detection, or general visual language models for chart understanding. However, Mermaid images contain small primitives, dense text, subtle arrow directions, and significant differences between solid and dashed lines. Furthermore, different chart categories exhibit substantial grammatical differences, making it difficult to recover complete structural relationships using only OCR and traditional image processing. While general multimodal models possess image understanding capabilities, they still suffer from instability in Mermaid category identification, grammatical constraints, complete node relationship reconstruction, and code renderability.

[0004] Furthermore, real-world Mermaid images often suffer from issues such as transparent backgrounds, light-colored lines, low resolution, small font sizes, and inconsistent margins. Directly inputting these into the model will degrade the quality of text and primitive recognition. Without targeted white background completion, compact cropping, and super-resolution enhancement, the model is prone to missing nodes, misjudging arrow endpoints, or generating unworkable Mermaid code.

[0005] Therefore, it is necessary to propose a multimodal large model fine-tuning method for Mermaid image parsing tasks, which combines self-synthesized supervised data, real image preprocessing, efficient LoRA parameter fine-tuning, and output syntax verification to improve the accuracy, stability, and feasibility of Mermaid image parsing. Summary of the Invention

[0006] This invention discloses a method for Mermaid image parsing and understanding based on LoRA fine-tuning of a multimodal large model. The method uses the Qwen3.5-27B multimodal large model as the base model and employs Unsloth for LoRA fine-tuning. The trained model directly generates structured JSON results from Mermaid images, including Mermaid image classification, Chinese descriptions, and valid Mermaid source code. This method forms a closed loop encompassing data synthesis, data formatting, image preprocessing, model fine-tuning, model inference, and result verification, enabling the model not only to recognize image content but also to output editable code conforming to Mermaid syntax.

[0007] This invention is achieved through the following technical solution: Step 1: Synthesize the Mermaid image dataset. Based on preset themes, chart types, and Mermaid syntax templates, generate Mermaid source code, render it as PNG or SVG images, and generate a Chinese description for each sample. Each sample contains fields such as png, svg, mmd_type_index, mmd_type_name, desc_zh_cn, and mmd. Step 2: Dataset Classification and Format Conversion. Read the label mapping from labels.json, split the training and test sets according to Mermaid type, forming train.json and test.json for each category; then convert the samples into ChatML-style multimodal supervised data, where user messages are task prompts, image fields are sample image paths, and assistant messages are JSON strings containing mmd_type_name, desc_zh_cn, and mmd. Step 3: Real Image Preprocessing. For images with transparent or no background, extract the visible area of ​​the Alpha channel, crop the transparent edges, and add a white background; for low-resolution images, load super-resolution models such as EDSR or FSRCNN using OpenCV dnn_superres for upsampling, and uniformly scale them to 2048 or 4096 for the long side to improve the recognizability of small text, arrows, and lines; Step 4: Efficient fine-tuning of LoRA parameters. Load the Qwen3.5-27B multimodal base model, accelerate training using Unsloth, and inject LoRA low-rank matrices into the linear modules of the attention layer and feedforward layer; the optimal parameters are r=8, lora_alpha=16, lora_dropout=0, bf16 precision, truncation length 2048, learning rate 5.0e-5, cosine scheduling, and 3 epochs; Step 5: Model Inference and Structured Output. After processing the Mermaid images to be parsed according to the same preprocessing procedure, input them along with the task prompts into the fine-tuning model. The model outputs JSON text; in the JSON text, mmd_type_name gives the image category, desc_zh_cn gives the Chinese semantic description, and mmd gives the renderable Mermaid code; Step Six: Post-processing and Validation. Perform JSON parsing, field checks, Markdown code block cleanup, and Mermaid keyword and syntax validation on the output. If the model output does not meet the structured requirements, trigger a retry or fall back to an "Unknown" result.

[0008] Furthermore, 11 high-frequency Mermaid categories were selected as the main training targets in the synthetic dataset, including Flowchart, Sequence Diagram, Class Diagram, Entity Relationship Diagram, UserJourney, Gantt, Pie Chart, Quadrant Chart, Git Diagram, Timeline, and XY Chart. One embodiment's synthetic-mmd-v2 dataset contains 14,322 training samples and 3,197 test samples, including 2,249 Flowchart training samples, 1,562 Git Diagram training samples, and approximately 1,128 to 1,200 training samples for the other categories.

[0009] Furthermore, the model output classification field is compatible with 22 Mermaid labels and an Unknown category, enabling the system to perform high-precision parsing on 11 high-frequency categories, as well as to perform a safe fallback when encountering untrained or uncertain chart types.

[0010] Furthermore, for the Class Diagram category, the prompts can be reinforced to emphasize the identification requirements of class structure, inheritance relationships, composition relationships, aggregation relationships, association relationships, arrow directions, and solid / dashed line types, enabling the model to perform a secondary check on connection types and arrow endpoints when generating Mermaid class diagram source code.

[0011] The beneficial effects of this invention are as follows: This invention solves the problem of high cost in Mermaid image annotation by using self-synthesized supervised data, improves the recognizability of real images through white background completion and super-resolution enhancement, enhances the task adaptability of Qwen3.5-27B to Mermaid syntax and chart structures with lower training cost through LoRA fine-tuning, and improves the usability of generated results through structured JSON output and Mermaid validation. Project evaluation records show that in the validated categories of Flowchart, GitDiagram, Pie Chart, Sequence Diagram, and User Journey, the LoRA model achieves an average improvement of approximately 49.10 in BLEU-4 and approximately 49.86 in ROUGE-L compared to the original model, indicating that this method can significantly enhance the ability to reconstruct Mermaid images from code. Attached Figure Description

[0012] To fully illustrate the technical solution of the present invention, the following will systematically present and analyze the selected embodiments through several accompanying drawings and corresponding textual content.

[0013] Figure 1 This is a schematic diagram of the overall process of the present invention; Figure 2 This is a schematic diagram of the self-synthesized Mermaid dataset construction process of the present invention; Figure 3 This is a schematic diagram of the actual Mermaid image preprocessing process of the present invention; Figure 4 This is a schematic diagram of the Unsloth LoRA fine-tuning process based on Qwen3.5-27B of the present invention; Figure 5 This is a schematic diagram of the model reasoning and structured output verification process of the present invention. Detailed Implementation

[0014] Please see Figures 1 to 5 As shown, this invention provides a method for Mermaid image parsing and understanding based on LoRA fine-tuning of a multimodal large model. In one specific embodiment, the project directory includes a basic model directory, a synthetic dataset directory, a classification formatting script, a real image preprocessing script, a LoRA training output directory, and an evaluation output directory, forming a complete engineering process from data construction to model validation.

[0015] S101: Constructing self-synthesized Mermaid image data. The system first generates or collects Mermaid source code and exports corresponding PNG and SVG images through the Mermaid renderer; then, it generates a Chinese description for each image, including the chart theme, nodes or entities, line direction, relationship type, time sequence, category proportion, or branch merging path. Each sample is saved in JSONL, with key fields including image path, SVG path, category number, category name, Chinese description, and Mermaid source code.

[0016] S102: Perform category splitting and multimodal instruction formatting. The system reads labels.json and establishes a mapping from category number to category name; it parses train.jsonl and test.jsonl line by line, categorizing them into the corresponding category directory according to mmd_type_name or mmd_type_index; each sample is converted into a training format containing messages and images fields, where the user content of messages is a fixed task prompt word, the assistant content is a JSON string, and the images array stores the image paths.

[0017] The task prompts require the model to act as a professional data visualization expert, outputting only JSON-formatted text based on the content of Mermaid images. The JSON contains three fields: mmd_type_name, desc_zh_cn, and mmd. When the category cannot be determined, the model should output "Unknown" and set desc_zh_cn and mmd to empty strings. When outputting Mermaid source code, the model must not include explanatory text or Markdown code blocks.

[0018] S103: Perform real-image preprocessing. For images with transparent backgrounds, the system converts the image to RGBA, reads the alpha channel, and generates a visible pixel mask according to a threshold. After obtaining the bounding box of the visible area, it performs compact cropping and then pastes the cropped result onto the RGB white canvas with preset padding. For low-resolution real-images, the OpenCV dnn_superres module is called to load the EDSR_x4 or FSRCNN_x4 model for super-resolution enhancement and scales it proportionally to a long side of 2048 or 4096.

[0019] Among them, super-resolution enhancement performs upsampling when the longer side of the input image is smaller than the longer side of the target image; during scaling, cubic interpolation is used for enlargement and area interpolation is used for shrinking, in order to maintain the proportion of the chart structure and enhance the visibility of small fonts, arrows and thin lines.

[0020] S104: Implement LoRA fine-tuning. The system loads the Qwen3.5-27B multimodal model along with its image processor and word segmenter, using Unsloth for training acceleration and memory optimization. With the basic model parameters frozen, LoRA parameters are injected into linear modules such as q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, in_proj_qkv, in_proj_a, in_proj_b, in_proj_z, and out_proj. Training preferably uses bf16, a batch size of 2 per device, 8 gradient accumulation steps, a learning rate of 5.0e-5, cosine scheduling, the AdamW optimizer, and a save step size of 100.

[0021] As one example, the system can use a unified LoRA adapter to complete the parsing of all categories, or it can train category-specific adapters separately. In the category-specific approach, the base model or lightweight classifier first determines the image category, and then loads the corresponding category's LoRA adapter to perform structured parsing for that category, thereby enhancing the ability to restore the source code of complex chart types.

[0022] S105: Execution Inference Generation. The system performs the same white background completion and super-resolution processing as in the training phase on the image to be parsed, and then inputs the image and task prompts into the fine-tuning model. The model generates JSON text based on the image's visual content and prompt constraints, where mmd_type_name is the category name, desc_zh_cn is the Chinese description, and mmd is the Mermaid source code.

[0023] S106: Perform structured output validation. The system cleans up redundant text in the output and parses the JSON; if necessary fields are missing, field types are incorrect, mmd_type_name is not in the tag set, or the Mermaid source code lacks the corresponding keyword for that category, the output is deemed unqualified. For unqualified results, the system can prompt for model regeneration; if it is still unqualified, an "Unknown" fallback result is output.

[0024] As an example, the system inputs test set images into the original model and the LoRA model, obtains the generated results, and calculates text similarity indicators such as BLEU-4, ROUGE-1, ROUGE-2 and ROUGE-L based on the reference Mermaid source code; it can also further perform Mermaid rendering verification to check whether the generated code can be successfully rendered into a chart.

[0025] As an example, the training / test distribution of the 11 classes of samples in the synthetic-mmd-v2 dataset is as follows: Flowchart 2249 / 475, Git Diagram 1562 / 391, Pie Chart 1200 / 263, User Journey 1187 / 223, Gantt 1186 / 248, Quadrant Chart 1168 / 264, XY Chart 1166 / 300, Sequence Diagram 1160 / 231, Entity Relationship Diagram 1159 / 287, Class Diagram 1157 / 234, and Timeline 1128 / 281. This distribution can cover the high-frequency Mermaid chart types found in common technical documents.

[0026] As an example, the LoRA adapter generated after training only saves the low-rank incremental weights, word segmenter configuration, processor configuration, and training state, without needing to copy the complete base model weights. This facilitates deployment and switching by category. During inference deployment, the base model is loaded once, and the system dynamically loads or merges the target LoRA adapter according to task requirements.

[0027] As an example, for the Class Diagram parsing task, the prompts further require the model to analyze the class structure, inheritance relationships, composition / aggregation relationships, association relationships, and arrow pointing directions, and require a secondary check to ensure that the arrow connection type and direction are consistent with the original... Figure 1 This approach reduces the likelihood of misjudging inheritance arrows, dependency dashed lines, composite diamond endpoints, and aggregate diamond endpoints in class diagrams.

[0028] As an example, for PNG images without a background in real-world images, the system first eliminates the influence of the alpha channel on the visual encoder by padding with a white background, then enhances the visibility of small fonts and fine lines through 2048 or 4096 long-side super-resolution, and finally inputs it into the model for parsing. This process is particularly suitable for Mermaid images obtained from web pages, Markdown documents, knowledge base screenshots, or exported files.

[0029] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, their core purpose is to illustrate the technical concept and feasible solutions of the present invention for ease of understanding. However, this is not intended to limit the scope of protection of the present invention. Based on the basic technical inspiration provided by the present invention, any logical extensions, modifications, or applications made by those skilled in the art without departing from the core idea of ​​the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for Mermaid image parsing and understanding based on LoRA fine-tuning of a multimodal large model, characterized in that, Includes the following steps: Step 1: Construct a Mermaid image training dataset, generate corresponding SVG or PNG images based on the Mermaid source code, and establish a mapping relationship between image path, Mermaid category, Chinese description and Mermaid source code for each sample, forming an image-description-code triple; Step 2: Perform category filtering, classification storage, and instruction formatting on the training dataset to convert the samples into multimodal supervised fine-tuning samples containing user prompts, image inputs, and assistant structured answers, wherein the assistant structured answers include three fields: mmd_type_name, desc_zh_cn, and mmd. Step 3: Perform preprocessing on the real Mermaid image to be analyzed, identify the transparency channel or missing background areas, crop the transparent edges based on the visible pixel bounding box, add a white background and preset margins around the primitives, and perform super-resolution enhancement according to the image resolution; Step 4: Load the Qwen3.5-27B multimodal base model, and use Unsloth to perform LoRA fine-tuning on the base model. Under the condition of freezing the base model parameters, insert low-rank adaptation parameters into the attention projection module, feedforward network module and multimodal projection related linear module. Step 5: Input the pre-processed Mermaid images and task prompts into the multimodal large model fine-tuned by LoRA to generate a parsing result containing only JSON text. In the JSON text, mmd_type_name represents the Mermaid image category, desc_zh_cn represents the Chinese chart description, and mmd represents the renderable Mermaid source code. Step Six: Perform structured parsing and validity checks on the model output, filter Markdown code blocks and explanatory text, verify the integrity of JSON fields and Mermaid source code syntax; when the image category cannot be determined or the source code is unrecoverable, set mmd_type_name to Unknown and set desc_zh_cn and mmd to empty strings.

2. The method as described in claim 1, characterized in that, The training dataset in step one is a self-synthesized dataset, which includes at least 11 types of Mermaid charts: Flowchart, Sequence Diagram, Class Diagram, Entity Relationship Diagram, User Journey, Gantt, Pie Chart, Quadrant Chart, Git Diagram, Timeline, and XYChart.

3. The method as described in claim 1, characterized in that, The data labeling system in step one is compatible with 22 label categories, including Unknown, Flowchart, Sequence Diagram, Class Diagram, State Diagram, Entity Relationship Diagram, User Journey, Gantt, Pie Chart, Quadrant Chart, Requirement Diagram, GitDiagram, Mindmaps, Timeline, ZenUML, XY Chart, Block Diagram, Packet, Kanban, Architecture, Radar, and Treemap. During training, the target high-frequency category is selected for fine-tuning.

4. The method as described in claim 1, characterized in that, In the image-description-code triple, the image field points to the PNG or SVG rendering result, the Chinese description field describes the chart title, nodes, lines, arrow direction, hierarchical structure, category proportion, time relationship or version branch relationship, and the source code field stores the Mermaid syntax text corresponding to the image.

5. The method as described in claim 1, characterized in that, The white background completion step three includes: converting the input image to RGBA format, extracting the alpha channel and obtaining the bounding box of visible pixels according to a threshold; after cropping the visible area, pasting the cropped image onto the RGB white canvas according to a preset padding to obtain an image with a pure white background.

6. The method as described in claim 1, characterized in that, The super-resolution enhancement in step three includes: calling the OpenCV dnn_superres module to load any super-resolution model from EDSR, FSRCNN, ESPCN, or LapSRN; performing upsampling when the longer side of the image is smaller than the longer side of the target image; and scaling the result to 2048 pixels or 4096 pixels according to the longer side of the target image.

7. The method as described in claim 1, characterized in that, The LoRA fine-tuning parameters in step four include: LoRA rank r = 8, lora_alpha = 16, lora_dropout = 0, training precision is bf16, sequence truncation length is 2048, optimizer is AdamW, learning rate is 5.0e-5, learning rate scheduling mode is cosine, and training epochs are 3.

8. The method as described in claim 1, characterized in that, In step four, the LoRA target module includes at least one or more of q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, in_proj_qkv, in_proj_a, in_proj_b, in_proj_z, and out_proj.

9. The method as described in claim 1, characterized in that, Step four also includes a category-specific adapter training strategy: LoRA adapters are trained separately for different Mermaid categories. During inference, the corresponding category adapter is selected first based on the coarse classification result of the image, and then the Chinese description and Mermaid source code under that category are generated.

10. The method as described in claim 1, characterized in that, In step five, the task prompt word constraint model only outputs JSON text and requires the model to identify class structure, inheritance relationship, composition relationship, aggregation relationship, association relationship, arrow endpoint direction, and solid or dashed line type in the class diagram parsing scenario; and, in step six, the legality verification includes JSON parsing verification, field set verification, Mermaid keyword verification, code block removal, null value fallback, and optional Mermaid rendering verification to improve the usability of the output code.

Images