Method and device for automatically generating mind map of handwritten conference record

Abstract

The invention relates to the technical field of natural language processing, and discloses a method and device for automatically generating a mind map of a handwritten conference record, and the method comprises the steps: obtaining a plurality of original handwritten conference record images of a target conference, and carrying out the multi-mode preprocessing, and obtaining a target handwritten conference record image; based on the target handwritten conference record image, a target JSON data set with coordinates is generated through recognition and decoding of a target OCR model and cooperative operation processing of a preset large language model, and then semantic understanding and processing are conducted through the preset large language model to obtain a target semantic block JSON array; performing deep semantic analysis on the conference content of the target conference by using the target semantic block JSON array, and generating a target tree structure JSON object; and the target mind map of the target conference is generated by using the target tree structure JSON object, so that the problems of flow splitting, lack of deep semantic understanding and poor fault tolerance and adaptability in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method and apparatus for automatically generating mind maps of handwritten meeting minutes. Background Technology

[0002] Current meeting minutes are mostly handwritten or computer-entered, resulting in unstructured, lengthy, and scattered content, making post-meeting organization time-consuming and labor-intensive. While existing Optical Character Recognition (OCR) technology can effectively convert printed text images into machine-coded text, its accuracy in recognizing handwritten text remains a challenge. Existing Large Language Modeling (LLM) technology can perform text summarization, keyword extraction, and entity recognition, but it is typically suited for clear, well-structured long texts. Direct application to short texts, which may contain errors, abbreviations, or grammatical inconsistencies after handwriting recognition, is ineffective. Furthermore, mind mapping tools require users to manually input and organize content, failing to achieve automatic conversion from raw handwritten materials to a finished product.

[0003] Existing technologies have the following drawbacks:

[0004] (1) Fragmented process: OCR, text cleaning, content understanding and mind map generation are separate steps, and no end-to-end automated process is formed.

[0005] (2) Lack of deep semantic understanding: Existing technologies have failed to effectively solve the problem of how to automatically extract the implicit logical hierarchy from unstructured, free-format handwriting. Visual-based methods lack semantic depth, while semantic-based LLM methods ignore the logical clues conveyed by the recorder through spatial layout.

[0006] (3) Poor fault tolerance and adaptability: By treating OCR and LLM as independent modules and combining them in series, information flows in one direction, and errors are propagated and amplified at each level, resulting in unusable final output results. Furthermore, it cannot adapt to different meeting types and recording styles, requiring a large amount of manual intervention and template configuration. Summary of the Invention

[0007] This invention provides a method and apparatus for automatically generating mind maps of handwritten meeting minutes, in order to solve the problems of fragmented processes, lack of deep semantic understanding, and poor fault tolerance and adaptability in existing technologies.

[0008] In a first aspect, the present invention provides a method for automatically generating mind maps of handwritten meeting minutes, the method comprising: The process involves: acquiring multiple original handwritten meeting transcript images of the target meeting; performing multimodal preprocessing on these images to obtain the target handwritten meeting transcript image; generating a target JSON dataset with coordinates through target OCR model recognition and decoding, and collaborative processing with a pre-defined large language model, with the target OCR model and pre-defined large language model operating collaboratively and integrated into a dual-engine module; performing semantic understanding and processing using the pre-defined large language model based on the target JSON dataset to obtain a target semantic block JSON array; performing deep semantic analysis on the meeting content of the target meeting using the target semantic block JSON array and generating a target tree-structured JSON object; and generating a target mind map of the target meeting using the target tree-structured JSON object.

[0009] This invention provides a method for automatically generating mind maps of handwritten meeting minutes. By acquiring multiple original handwritten meeting minutes images of the target meeting, it can collect complete original meeting data, avoiding information omissions due to data gaps. Furthermore, by performing multimodal preprocessing on multiple original handwritten meeting minutes images, it can eliminate image noise, distortion, and tilt, unifying image format and quality, and improving basic recognition accuracy. Further, by generating a target JSON dataset through target OCR model recognition and decoding and pre-set large language model processing, it realizes the transformation of handwritten text from image to structured text. Simultaneously, through dual-engine collaborative error correction, it ensures the accuracy and structure of the text data, providing a reliable foundation for semantic processing. Further, by using the pre-set large language model for semantic understanding and processing, it can integrate spatial relationship features and generate high-quality semantic blocks. Further, by using the target semantic block JSON array, it performs deep semantic analysis of the meeting content and generates a target tree-structured JSON object, integrating semantic and visual layout information. This allows for accurate reconstruction of the logical hierarchy and relationships of the meeting content, forming a structured blueprint for the mind map. Furthermore, by using the target tree structure JSON object to generate target mind maps, structured data can be transformed into intuitive and interactive visualizations, enabling efficient presentation and utilization of meeting information and helping to support meeting debriefing, task tracking, and knowledge accumulation.

[0010] In one optional implementation, multimodal preprocessing is performed on multiple original handwritten meeting record images to obtain a target handwritten meeting record image, including: Multiple original handwritten meeting record images are stitched and registered to obtain the first handwritten meeting record image; the first handwritten meeting record image is enhanced to obtain the second handwritten meeting record image; the second handwritten meeting record image is geometrically corrected to obtain the third handwritten meeting record image; the text region of the third handwritten meeting record image is located and cropped using a text detection model to obtain the target handwritten meeting record image.

[0011] The method for automatically generating mind maps of handwritten meeting minutes provided by this invention integrates multi-source image data through stitching and registration processing, ensuring complete coverage of the text area and avoiding information breaks caused by image fragmentation, thus guaranteeing data integrity. Furthermore, by enhancing the first handwritten meeting minute image, image noise interference is reduced, highlighting the contrast between the text area and the background, improving text legibility and reducing the difficulty of subsequent recognition. Furthermore, geometric correction corrects image tilt, ensuring text lines remain horizontal and standardized, reducing geometric interference during OCR recognition and improving character recognition accuracy. Furthermore, text detection model processing accurately locates the text area, narrowing the OCR processing range, reducing irrelevant background interference, and improving recognition efficiency and accuracy.

[0012] In one optional implementation, based on the target handwritten meeting transcript image, after recognition and decoding by a target OCR model and collaborative processing by a preset large language model, a target JSON dataset with coordinates is generated, including: Based on the target handwritten meeting transcript image, text detection and recognition are performed using a target OCR model to obtain an initial probability matrix of multiple characters. Based on the initial probability matrix, multiple confidence values ​​of multiple characters in each path of the target candidate path set are calculated. The target candidate path set is obtained by expanding each path in the preset candidate path set. When multiple confidence values ​​are greater than a preset confidence threshold, the initial candidate path set is updated using the multiple confidence values ​​and the target candidate path set, and multiple target text lines are determined. Based on the multiple target text lines, the coordinate information of the multiple target text lines in the target handwritten meeting transcript image, and the multiple confidence values, a target JSON dataset with coordinates is generated.

[0013] The method for automatically generating mind maps of handwritten meeting minutes provided by this invention utilizes a target OCR model for text detection and recognition, converting image text into machine-processable character probability information. Furthermore, by calculating confidence values, the reliability of character recognition is quantified, providing a basis for judging recognition accuracy and triggering error correction mechanisms. Further, when multiple confidence values ​​are greater than a preset confidence threshold, the initial candidate path set is updated and multiple target text lines are determined, quickly locking in high-confidence, accurate recognition results and ensuring the basic reliability of the text data. Furthermore, a target JSON dataset is generated based on multiple target text lines, coordinate information, and multiple confidence values, achieving a structured integration of text content, spatial location, and confidence, providing standardized input containing multi-dimensional information for subsequent semantic processing.

[0014] In one optional implementation, based on the target handwritten meeting transcript image, a target OCR model is used for text detection and recognition to obtain an initial probability matrix of multiple characters, including: The target handwritten meeting transcript image is input into the target OCR model to obtain multiple text line bounding boxes. Based on the multiple text line bounding boxes, multiple initial text line image patches are determined in the target handwritten meeting transcript image. The multiple initial text line image patches are scaled to obtain multiple target text line image patches. The multiple target text line image patches are input into the CNN backbone network for processing to determine the initial feature sequence. The initial feature sequence is input into the Transformer encoder for processing to obtain the target feature sequence. The target feature sequence is processed through a fully connected layer and a Softmax function to generate an initial probability matrix.

[0015] This invention provides a method for automatically generating mind maps of handwritten meeting minutes. By inputting the target handwritten meeting minute image into a target OCR model, multiple text line bounding boxes are obtained, accurately defining the text line range and providing precise positioning for subsequent image block cropping and recognition. Furthermore, multiple initial text line image blocks are determined based on the multiple text line bounding boxes, separating individual text lines, avoiding interference between text lines, and improving recognition targeting. Furthermore, scaling the multiple initial text line image blocks unifies the image size standard, thereby adapting to the input requirements of the OCR model and ensuring the consistency and stability of model processing. Furthermore, inputting the multiple target text line image blocks into a CNN backbone network allows for the extraction of visual features from the text image. Furthermore, inputting the initial feature sequence into a Transformer encoder for processing yields the target feature sequence, strengthening global context modeling, improving the semantic relevance of the feature sequence, and contributing to improved coherence and accuracy of character recognition. Furthermore, processing the target feature sequence through a fully connected layer and a Softmax function respectively transforms the feature information into a character probability distribution, thus providing a quantitative basis for subsequent decoding of the text string.

[0016] In one alternative implementation, the method further includes: When the confidence score of a target character in any path of the target candidate path set is less than a preset confidence threshold, multiple identified characters are extracted from the corresponding target path. Based on the bounding boxes of the text lines corresponding to these identified characters and the position of the target character, target description information is generated. This target description information is then input into a preset large language model for processing, yielding multiple candidate words and their relevance scores. Based on these relevance scores, normalized scores for candidate words containing the target character are calculated. The normalized scores and the original probability values ​​of the target character are weighted and fused to obtain the target probability value. The target probability value is then used to update the original probability values ​​in the initial probability matrix, resulting in the target probability matrix. Based on the target probability matrix, the process of calculating multiple confidence scores is repeated iteratively until multiple target text lines are obtained.

[0017] The method for automatically generating mind maps for handwritten meeting minutes provided by this invention extracts identified characters and generates target description information when the confidence level of a target character in any path within the target candidate path set is less than a preset confidence threshold. This accurately locates low-confidence identification nodes, providing clear context and location information for LLM error correction. Furthermore, the target description information is input into a preset large language model for processing to obtain multiple candidate words and relevance scores. By utilizing the semantic understanding capabilities of LLM, highly relevant candidate answers are provided for low-confidence characters, providing a reliable basis for error correction. Further, based on multiple relevance scores, the normalized scores of candidate words containing the target character are calculated, quantifying the relative importance of candidate words and providing standardized data for probability fusion. Finally, the normalized scores and the original probability values ​​of the target characters are weighted and fused to obtain the target probability value. This combines visual recognition evidence and semantic reasoning results, optimizing the character probability distribution and improving the recognition accuracy of low-confidence characters. Furthermore, by updating the initial probability matrix using the target probability value and iterating repeatedly until multiple target text lines are obtained, error propagation is blocked, the recognition results can be continuously corrected, the recognition accuracy of handwritten text is significantly improved, and the fault tolerance is enhanced.

[0018] In one optional implementation, based on the target JSON dataset, semantic understanding and processing are performed using a pre-defined large language model to obtain a target semantic block JSON array, including: The target JSON dataset is parsed to generate a set of text blocks containing spatial context; a first Prompt is constructed based on the set of text blocks; the Prompt is processed by a pre-defined large language model to obtain a target semantic block JSON array.

[0019] The method for automatically generating mind maps for handwritten meeting minutes provided by this invention parses a target JSON dataset and generates a set of text blocks containing spatial context. This process uncovers spatial relationship features between text blocks, providing crucial clues for multimodal semantic understanding and facilitating accurate grasp of textual logical connections. Furthermore, a first Prompt is constructed based on the text block set, integrating multi-dimensional information such as text content and spatial relationships into instructions understandable by a pre-defined large language model, thus guiding the model to efficiently complete multi-task processing. Further, the Prompt is processed by the pre-defined large language model to obtain a target semantic block JSON array, achieving integrated processing of text cleaning and error correction, entity extraction, and hierarchical prediction. This results in the generation of high-quality semantic blocks with complete structure and rich information, laying a solid foundation for deep structure analysis.

[0020] In one optional implementation, a deep semantic parsing of the meeting content of the target meeting is performed using a target semantic block JSON array, and a target tree-structured JSON object is generated, including: Global information integration is performed on the target semantic block JSON array to obtain a multimodal context information set; the multimodal context information set is input into the second Prompt to obtain a standard nested JSON structure dataset, wherein the second Prompt is generated according to content belonging determination rules based on semantic relevance and visual layout; based on the multimodal context information set and the standard nested JSON structure dataset, topics are automatically summarized and grouped to obtain an initial tree-structured JSON object; the initial tree-structured JSON object is structurally optimized and conflict resolved to obtain the target tree-structured JSON object.

[0021] This invention provides a method for automatically generating mind maps for handwritten meeting minutes. It integrates multimodal contextual information from a target semantic block JSON array to obtain a multimodal contextual information set, incorporating multi-source information such as text semantics, visual layout, and semantic association matrices, providing comprehensive and rich contextual support for logical structure reasoning. Furthermore, the multimodal contextual information set is input into a second Prompt to obtain a standard nested JSON structure dataset. LLM reasoning capabilities and preset attribution rules are utilized to initially construct hierarchical relationships between text blocks, forming a basic tree structure. Further, based on the multimodal contextual information set and the standard nested JSON structure dataset, topics are automatically summarized and grouped to obtain an initial tree structure JSON object. Through clustering algorithms and LLM topic generation, similar content is categorized and summarized, making the tree structure more logical and concise. Finally, the initial tree structure JSON object is structurally optimized and conflict-resolved to obtain a target tree structure JSON object. This eliminates logical conflicts such as circular structures and hierarchical jumps, optimizes node sorting and attribution relationships, and ensures the standardization, rationality, and rigor of the tree structure, providing a high-quality blueprint for mind map generation.

[0022] In one alternative implementation, a target mind map of the target meeting is generated using a target tree-structured JSON object, including: Render the target tree-structured JSON object to obtain the initial mind map component; add a click event to the initial mind map component to obtain the target mind map component; transform the elements in the target mind map component to obtain the target mind map.

[0023] This invention provides a method for automatically generating mind maps for handwritten meeting minutes. It renders a target tree-structured JSON object to obtain an initial mind map component, transforming structured data into a visual graphic and providing an intuitive presentation of the meeting's logical structure. Furthermore, by adding click events to the initial mind map component, a target mind map component is obtained, endowing the mind map with interactive functionality and supporting node expansion / collapse, thus improving user convenience and information retrieval flexibility. Moreover, by transforming the elements in the target mind map component, an exportable PNG / SVG format image can be generated, facilitating user saving, sharing, and use, maximizing the utilization value of meeting information.

[0024] Secondly, the present invention provides an apparatus for automatically generating mind maps of handwritten meeting minutes, the apparatus comprising: The module comprises three parts: an acquisition module for acquiring multiple original handwritten meeting transcript images of the target meeting; a first processing module for performing multimodal preprocessing on the multiple original handwritten meeting transcript images to obtain the target handwritten meeting transcript image; a second processing module for generating a target JSON dataset with coordinates based on the target handwritten meeting transcript image through recognition and decoding by the target OCR model and collaborative processing by a preset large language model, with the target OCR model and the preset large language model running collaboratively and integrated in a dual-engine module; a third processing module for performing semantic understanding and processing based on the target JSON dataset using the preset large language model to obtain a target semantic block JSON array; a parsing module for performing deep semantic parsing of the meeting content of the target meeting using the target semantic block JSON array and generating a target tree-structured JSON object; and a generation module for generating a target mind map of the target meeting using the target tree-structured JSON object.

[0025] Thirdly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the method for automatically generating mind maps of handwritten meeting minutes according to the first aspect above or any corresponding embodiment. Attached Figure Description

[0026] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of an application scenario according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a method for automatically generating mind maps of handwritten meeting minutes according to an embodiment of the present invention. Figure 3 This is a flowchart of the multimodal image preprocessing and neighborhood-adaptive OCR recognition module according to an embodiment of the present invention; Figure 4 This is a flowchart of the LLM-driven semantic understanding and structured cleaning module according to an embodiment of the present invention; Figure 5 This is a structural block diagram of an apparatus for automatically generating mind maps of handwritten meeting minutes according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0030] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0031] As an optional application scenario of this invention, the specific application environment architecture or specific hardware architecture on which the method for automatically generating mind maps of handwritten meeting minutes depends is described herein. For example... Figure 1 As shown, the architecture system may include at least one terminal device and at least one server. Figure 1 The system is illustrated in the example, which includes a computer 101, a mobile terminal 102, and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0032] Specifically, the terminal device can be a smartphone, tablet, laptop, PDA, desktop computer, game console, smart TV, smart wearable device, in-vehicle terminal, VR (Virtual Reality) device, AR (Augmented Reality) device, etc. Server 103 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. Network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranet, local area network, wide area network, mobile communication network, and combinations thereof.

[0033] This invention provides a method for automatically generating mind maps of handwritten meeting minutes. By using a dual-engine collaborative processing of an OCR model and a preset large language model, the handwritten meeting minutes are automatically transformed into an intuitive, interactive, and visual mind map.

[0034] According to an embodiment of the present invention, a method for automatically generating mind maps of handwritten meeting minutes is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] This embodiment provides a method for automatically generating mind maps of handwritten meeting minutes, which can be used on the aforementioned mobile terminals, such as mobile phones and tablets. Figure 2 This is a flowchart of a method for automatically generating mind maps of handwritten meeting minutes according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain multiple original handwritten meeting record images of the target meeting.

[0036] In one optional embodiment, the original handwritten meeting record image represents the image corresponding to the handwritten carrier that recorded the core information of the meeting (topics, key points, discussion conclusions, task assignments, etc.) through various acquisition devices during the target meeting. It can be a large amount of handwritten meeting record data from different sources, such as whiteboard photos containing handwritten meeting records, scanned copies of paper notes, and electronic handwriting input.

[0037] In one alternative embodiment, the handwritten media for meeting minutes can be diverse, such as whiteboards, paper notes, electronic writing boards, etc. A single media may result in information omissions (such as supplementary notes from different participants, or incomplete recordings of whiteboard content).

[0038] Therefore, this embodiment, by acquiring multiple original images, can cover all the core information of the meeting minutes. Simultaneously, the correlation between multiple source images provides a foundation for subsequent image stitching and registration, ensuring that the final integrated text area is complete and without gaps, and avoiding the impact of data defects from a single image on subsequent processing results.

[0039] Step S202: Perform multimodal preprocessing on multiple original handwritten meeting record images to obtain the target handwritten meeting record image.

[0040] In one optional embodiment, the original handwritten meeting record images acquired from multiple sources and without any processing are processed by multimodal specialized processing operations adapted to the meeting scenario. This process eliminates interference factors such as noise, distortion, and tilt generated during image acquisition, thereby integrating scattered text regions, extracting core text-related regions, and finally outputting a standardized image of uniform quality, clear text, and suitable for subsequent OCR recognition, namely the target handwritten meeting record image.

[0041] Step S203: Based on the target handwritten meeting record image, the target OCR model recognizes and decodes the image, and the preset large language model processes the data to generate a target JSON dataset with coordinates.

[0042] In one alternative embodiment, the target OCR model and the preset large language model work together and are integrated in a dual-engine module.

[0043] In one optional embodiment, the target OCR (Optical Character Recognition) model represents an optical character recognition model that has undergone domain-specific fine-tuning for handwritten meeting records, enabling it to be adapted to recognition scenarios of handwritten meeting record images from multiple sources, such as whiteboard photos, paper notes, and electronic writing tablets.

[0044] Furthermore, the target OCR model can optimize the font style, abbreviation habits, and professional terminology of handwritten meeting minutes, thereby accurately extracting text regions and converting them into character information. It also has the interactive capability of real-time collaborative error correction with the preset Large Language Model (LLM), rather than a traditional independent OCR tool.

[0045] In one optional embodiment, the pre-defined Large Language Model (LLM) represents a deep learning model trained with a large amount of text data, capable of generating natural language text or understanding language text, and possessing powerful semantic understanding, contextual reasoning, candidate word generation, and error correction capabilities.

[0046] Furthermore, in this embodiment, the preset large language model and the target OCR model form a dual-engine collaborative architecture. When the confidence level of the OCR recognition result is lower than the threshold, highly relevant candidate words are provided based on information such as meeting context and text location, which in turn guides the OCR to optimize the recognition result. At the same time, it participates in text semantic verification and probability fusion, which can ensure the accuracy of the recognition result.

[0047] In one optional embodiment, the target JSON dataset represents a structured data file that integrates the core information of the handwritten meeting minutes. It may include the specific content of each piece of recognized text, the coordinate information (bounding box) of the text in the original target handwritten meeting minutes image, the confidence score of the recognition result, and other information.

[0048] In one optional embodiment, a dual-engine architecture based on domain-adaptive OCR and a pre-defined large language model is used. The pre-processed standardized target handwritten meeting record image is used as input. The visual perception capability of the OCR model is used to complete the initial text recognition and probability quantization. The semantic cognition advantage of the large language model is combined to perform real-time error correction and probability fusion on the low confidence recognition results. At the same time, multi-dimensional information such as text content, spatial coordinates and recognition confidence are integrated to transform the unstructured handwritten image text into a structured target JSON dataset.

[0049] Step S204: Based on the target JSON dataset, use a pre-defined large language model to perform semantic understanding and processing to obtain a target semantic block JSON array.

[0050] In one optional embodiment, the target semantic block JSON array represents a standardized structured data set that integrates multi-dimensional high-quality information after deep processing by a preset large language model.

[0051] Furthermore, each array element, i.e. a semantic block, corresponds to an independent text unit in the meeting minutes and may contain text block ID, original text, final text after cleaning and error correction, estimated level (such as title, subtitle, body text), list of identified entities (names, time, topics, etc.), core keywords, comprehensive confidence (integrating OCR recognition confidence and LLM processing confidence), original coordinate information, and spatial relationship features, etc.

[0052] In one optional embodiment, based on the core logic of multimodal information fusion and semantic depth addition, and with a preset large language model as the intelligent hub, the association value between the semantics of text content and spatial layout clues is fully explored, thereby enabling the quality improvement and information enrichment of the preliminary structured data.

[0053] Specifically, by leveraging the powerful contextual understanding and error correction capabilities of large language models, we can correct potential issues in the target JSON dataset, such as recognition errors, grammatical inconsistencies, and missing abbreviations. At the same time, we can accurately extract entities and keywords to assign semantic labels to the text.

[0054] Furthermore, the coordinate information of text blocks is transformed into spatial relationship features that can be understood by machines, and used together with the text content as input to guide the large language model to predict the text hierarchy by combining visual layout cues, thereby achieving a deep integration of semantic and visual information.

[0055] Furthermore, by integrating OCR recognition confidence with LLM self-evaluation confidence, data reliability is quantified, thereby forming a standardized semantic block array, namely the target semantic block JSON array.

[0056] Step S205: Using the target semantic block JSON array, perform deep semantic parsing on the meeting content of the target meeting and generate a target tree structure JSON object.

[0057] In one alternative embodiment, the target tree structure JSON object represents standardized structured data that accurately describes the logical hierarchical relationship of the target meeting content, and is also a direct blueprint for generating mind maps.

[0058] Furthermore, the target tree-structured JSON object is in a nested tree format, with each node corresponding to a core information unit in the meeting (such as topic, key points, sub-items, etc.), including node content, attributes (such as entity, keywords, confidence level) and hierarchical relationships with other nodes (parent node, child node), which can completely and clearly restore the deep logical structure of the meeting content.

[0059] In one optional embodiment, the target semantic block JSON array containing multi-dimensional information such as text content, spatial relationships, entities, keywords, and hierarchical prediction is used as input. By integrating text semantics and visual layout information for joint reasoning, the logical relationship mining, topic summarization and grouping, and structural optimization of the meeting content are completed. Finally, the scattered semantic blocks are reconstructed into a complete, standardized, and logically rigorous tree-structured JSON object.

[0060] Step S206: Use the target tree structure JSON object to generate a target mind map for the target meeting.

[0061] In one alternative embodiment, the target mind map represents a visual and interactive presentation carrier of meeting information generated based on a target tree structure JSON object.

[0062] Furthermore, this mind map presents the core content of the meeting in a tree-like hierarchical structure, clearly showing the logical connections between topics, key points, and sub-items. It supports interactive operations such as expanding / collapsed nodes and can be exported as PNG or SVG format images, directly serving meeting debriefing, task tracking, and knowledge accumulation.

[0063] In one optional embodiment, based on the target tree-structured JSON object, the structured tree data is transformed into an intuitive, interactive, and exportable target mind map through visual rendering, interactive function binding, and format conversion. This realizes the transformation from structured digital data to a visual and practical tool, thereby making the meeting logic and core information easier to understand, use, and disseminate.

[0064] The method for automatically generating mind maps of handwritten meeting minutes provided in this embodiment collects complete original meeting data by acquiring multiple original handwritten meeting minutes images of the target meeting, avoiding information omissions due to data gaps. Furthermore, multimodal preprocessing of these images eliminates image noise, distortion, and tilt, unifying image format and quality and improving basic recognition accuracy. Further, by using a target OCR model for recognition and decoding, and a preset large language model for processing, a target JSON dataset is generated, realizing the transformation of handwritten text from image to structured text. Simultaneously, dual-engine collaborative correction of recognition errors ensures the accuracy and structure of the text data, providing a reliable foundation for semantic processing. Further, semantic understanding and processing using a preset large language model integrates spatial relationship features and generates high-quality semantic blocks. Finally, using a target semantic block JSON array, deep semantic analysis of the meeting content is performed, generating a target tree-structured JSON object that integrates semantic and visual layout information. This allows for accurate reconstruction of the logical hierarchy and relationships of the meeting content, forming a structured blueprint for the mind map. Furthermore, by using the target tree structure JSON object to generate target mind maps, structured data can be transformed into intuitive and interactive visualizations, enabling efficient presentation and utilization of meeting information and helping to support meeting debriefing, task tracking, and knowledge accumulation.

[0065] In some optional implementations, step S202 above includes: Step S2021: Perform stitching and registration processing on multiple original handwritten meeting record images to obtain the first handwritten meeting record image.

[0066] In one optional embodiment, the original handwritten meeting record images may come from different acquisition angles and different carriers, resulting in problems such as scattered text areas, overlapping or misaligned content. Therefore, in this embodiment, feature matching and spatial coordinate calibration techniques are used to integrate all images into a unified image that completely covers the meeting text area, has no information breaks, and is positionally aligned, i.e., the first handwritten meeting record image.

[0067] In one optional embodiment, by extracting key feature points from each image, using a feature matching algorithm to find the corresponding relationship between images, and then using the Homography matrix for spatial coordinate transformation and registration, the positional deviation between images can be eliminated. Finally, multiple images are stitched together to ensure that all meeting text areas are completely connected, thereby forming a unified image with comprehensive information and accurate location, namely the first handwritten meeting record image.

[0068] For example, for each original handwritten meeting record image, feature detection algorithms (such as SIFT and SURF) are used to extract key feature points and their descriptors in the image. These feature points can uniquely identify key areas such as text edges and character outlines in the image.

[0069] Furthermore, feature matching algorithms (such as the FLANN matcher) are used to compare feature point descriptors between different images and find corresponding feature point pairs between images, thereby determining the overlapping areas and positional associations between images.

[0070] Furthermore, based on the matched feature point pairs, the Homography matrix H is calculated. This matrix is ​​then used to perform spatial coordinate transformation on each image, correcting positional deviations such as scaling, rotation, and translation between images, thereby achieving precise alignment of all images in the same coordinate system.

[0071] Furthermore, the registered images are fused and stitched together, and an image fusion algorithm (such as linear fusion) is used to process the pixel transitions in the overlapping areas to avoid stitching marks, thereby ultimately generating a first handwritten meeting record image that covers all meeting text areas.

[0072] Step S2022: Enhance the first handwritten meeting record image to obtain the second handwritten meeting record image.

[0073] In one optional embodiment, irrelevant noise is first filtered out using a denoising algorithm, preserving the core features of the text. Then, binarization processing is used to convert the grayscale image into a black and white binary image, maximizing the contrast between the text and the background, making the character outlines clearer, and obtaining the processed secondhand meeting transcript image.

[0074] For example, firstly, a nonlocal means denoising algorithm is used to process the first handwritten meeting transcript image. The nonlocal means denoising algorithm effectively suppresses Gaussian noise, salt-and-pepper noise, etc., by finding similar pixel blocks in the image and performing a weighted average, while preserving the edge and detail information of the text, thus avoiding the loss of key character features during the denoising process.

[0075] Secondly, the contrast of the denoised image is adjusted, and the grayscale distribution of the image is optimized by histogram equalization algorithm to improve the brightness difference between the text area and the background.

[0076] Finally, an adaptive thresholding method is used to binarize the image, and the threshold is dynamically adjusted according to the local grayscale features of different regions of the image. Text areas above the threshold are set to white, and background areas below the threshold are set to black, thereby generating a second handwritten meeting record image with prominent text and a clean background.

[0077] Step S2023: Perform geometric correction on the second handwritten meeting record image to obtain the third handwritten meeting record image.

[0078] In one alternative embodiment, by detecting the tilt angle of the text lines and making targeted corrections, a target handwritten meeting record image with horizontally aligned text lines and no geometric deviations can be generated.

[0079] For example, the Hough transform algorithm can be used to detect text lines in a second-hand meeting transcript image. By identifying the straight-line features of the text lines, the tilt angle between the text lines and the horizontal direction can be calculated. .

[0080] Furthermore, based on the detected tilt angle The image is rotated in the reverse direction; if the image is tilted clockwise... If the angle is counterclockwise, rotate counterclockwise. Angles, and vice versa, to ensure that text lines are consistent with the horizontal direction.

[0081] Step S2024: Use a text detection model to locate and crop the text region of the third handwritten meeting record image to obtain the target handwritten meeting record image.

[0082] In one alternative embodiment, the text detection model represents an artificial intelligence model trained using a deep learning architecture (such as a convolutional neural network, Transformer, etc.) to automatically identify and locate text regions from an image.

[0083] In one alternative embodiment, the target handwritten meeting record image represents an independent image patch that is precisely cropped from the third handwritten meeting record image, based on the bounding box coordinates output by the text detection model, containing only the text region and excluding irrelevant background.

[0084] In one optional embodiment, the third handwritten meeting record image generated by rotation correction may have blank areas or incomplete cropping at the image edges. Invalid blank areas at the edges can be removed by a cropping algorithm to retain the complete text area, and finally generate a geometrically normalized and clear target handwritten meeting record image.

[0085] In one optional embodiment, a preprocessed, geometrically regular, and clearly written third-party handwritten meeting record image is used as input. Through a text detection model, the regions containing all handwritten text in the image can be accurately located. These regions are then extracted and cropped separately to form multiple images of interest (ROIs) that contain only the text content and exclude irrelevant background interference, i.e., the target handwritten meeting record image.

[0086] For example, first select a deep learning-based text detection model (such as the optimized EAST model) that has been fine-tuned with a dataset of handwritten meeting transcript images, load the pre-trained weights and configuration parameters, and ensure that the model is adapted to the text layout and features of the handwritten meeting transcripts.

[0087] Secondly, the target handwritten meeting record image is scaled to the input size required by the model (while maintaining the aspect ratio) and normalized to convert it into a tensor format that the model can recognize.

[0088] Then, the preprocessed image is input into the text detection model. The model, through processes such as feature extraction, multi-scale fusion, and bounding box prediction, outputs the bounding box coordinates of all text regions in the image. ).in, and These represent the minimum and maximum x-coordinates of the bounding box (the rectangle enclosing the text line) in the horizontal direction (X-axis), respectively. and These represent the minimum and maximum ordinates of the bounding box (the rectangle surrounding the text line) in the vertical direction (Y-axis), respectively.

[0089] Finally, based on the bounding boxes of each text region output by the model, the corresponding regions can be precisely cropped from the original target handwritten meeting record image, resulting in multiple independent images of interest (i.e., target handwritten meeting record images), while retaining the bounding box coordinate information corresponding to the images of interest. Each image of interest contains only a single continuous segment of text.

[0090] In some optional implementations, step S203 above includes: Step S2031: Based on the target handwritten meeting record image, use the target OCR model to perform text detection and recognition to obtain an initial probability matrix of multiple characters.

[0091] In one optional embodiment, multiple images of interest are used as input. The target OCR model, which has been finely tuned for the handwritten conference font domain, first accurately locates the text line boundaries in each image of interest again, then converts the text line images into character-level feature representations, and finally calculates and outputs the probability distribution of all possible characters corresponding to each position through the model, forming the corresponding initial probability matrix.

[0092] Specifically, step S2031 includes: Step a1: Input the target handwritten meeting record image into the target OCR model to obtain multiple text line bounding boxes.

[0093] Step a2: Based on multiple text line bounding boxes, determine multiple initial text line image blocks in the target handwritten meeting record image.

[0094] Step a3: Scale the multiple initial text line image blocks to obtain multiple target text line image blocks.

[0095] Step a4: Input multiple target text line image blocks into the CNN backbone network for processing and determine the initial feature sequence.

[0096] Step a5: Input the initial feature sequence into the Transformer encoder for processing to obtain the target feature sequence.

[0097] Step a6: The target feature sequence is processed through a fully connected layer and a Softmax function, and an initial probability matrix is ​​generated.

[0098] In an alternative embodiment, the Convolutional Neural Network Backbone (CNN) represents the component in the target OCR model responsible for extracting visual features of the image.

[0099] In one alternative embodiment, the Transformer encoder represents a feature enhancement component based on a self-attention mechanism for capturing global contextual relationships in sequence data.

[0100] In an alternative embodiment, the fully connected layer represents a key component for feature mapping and dimension transformation, located after the Transformer encoder and before the Softmax function.

[0101] In an alternative embodiment, the Softmax function represents a probability normalization function used to convert the character score vector output by the fully connected layer into a probability distribution.

[0102] In one optional embodiment, the target OCR model is specifically optimized for the font style and stroke features of handwritten text, effectively capturing the visual information of handwritten text. First, text regions are further refined through text line detection to ensure that each text line is processed independently. Then, local visual features of the text line image are extracted through a CNN backbone network to form an initial feature sequence. Subsequently, the global modeling capability of the Transformer encoder is used to strengthen the contextual association between characters (such as handwritten cursive, word collocation, etc.). Finally, through fully connected layer mapping and Softmax function normalization, the feature sequence is transformed into the probability distribution of all characters at each position, generating an initial probability matrix, thus realizing the transformation from image to character probability.

[0103] For example, the target handwritten meeting transcript image can be formatted according to the input requirements of the target OCR model, such as converting RGB to grayscale and adjusting the channel order. Furthermore, the text line detection submodule (based on a deep learning architecture) in the target OCR model can be called to scan each image of interest in the target handwritten meeting transcript image, identify continuous text line regions, and output the bounding box coordinates of each text line. This ensures that individual text lines are accurately selected. Furthermore, it allows for the collection of bounding boxes for all text lines corresponding to all images of interest, assigning a unique identifier to each bounding box and associating it with the source of the image of interest. The bounding box coordinates are the pixel positions within the original image.

[0104] Furthermore, the coordinates of the text line bounding boxes are mapped back to the coordinate system of the original target handwritten meeting record image to ensure precise positional correspondence. Then, based on the mapped bounding box coordinates, the region corresponding to each text line is cropped from the target handwritten meeting record image, resulting in multiple initial text line image blocks. Furthermore, each image block contains only one complete line of handwritten text.

[0105] Furthermore, invalid image blocks that are too small or have too low a text ratio after cropping can be removed, while valid initial text line image blocks can be retained.

[0106] Furthermore, a fixed input size is set for the target OCR model, and each initial text line image block is scaled using a bilinear interpolation algorithm to ensure that the text features of the scaled image are not distorted, while maintaining the aspect ratio of the text lines to fit the model input. Additionally, all scaled text line image blocks are uniformly named and stored to form multiple target text line image blocks, ensuring that the model can process them in batches.

[0107] Furthermore, a pre-trained CNN backbone network (such as ResNet or MobileNet) is loaded into the target OCR model. This CNN backbone network has learned general visual features through a large amount of image data and has been fine-tuned with handwritten conference data.

[0108] Furthermore, each target text line image block is input into the CNN backbone network, and through operations such as convolution and pooling, low-level visual features (such as stroke edges and textures) and high-level semantic features (such as character structure) of the text are gradually extracted, and then the corresponding two-dimensional feature map is output. Furthermore, the two-dimensional feature map is subjected to global average pooling along the height dimension to compress the height dimension information, and then expanded along the width dimension to form a one-dimensional initial feature sequence. ,in, For sequence length, , For feature dimensions.

[0109] Furthermore, the Transformer encoder incorporates a multi-layer self-attention mechanism and a feedforward neural network. Therefore, when the initial feature sequence F is input into the Transformer encoder, the self-attention mechanism captures long-distance dependencies between characters by calculating the association weights between each position within the sequence and all other positions. Further, after multi-layer Transformer encoding, the contextual information of the initial feature sequence is enhanced, and an enhanced target feature sequence is output. .in .

[0110] Furthermore, the target feature sequence Each feature vector in Input to a fully connected layer, the weight matrix of that layer .in, The size of the character table, containing all possible characters and whitespace, will be... 3D feature vectors are mapped to A character score vector of dimension.

[0111] Furthermore, for each position Applying the Softmax function to the score vector transforms the score into a probability value between 0 and 1, with the sum of the probabilities of all characters being 1, and yields the probability distribution of the character at that position: .in, It is the bias vector.

[0112] Furthermore, the character probability distributions at all positions are integrated to form an initial probability matrix. , of which elements Indicates at time step (Corresponding to a specific position in the text line), character probability .

[0113] Step S2032: Based on the initial probability matrix, calculate multiple confidence values ​​for multiple characters of each path in the target candidate path set.

[0114] In one alternative embodiment, the target candidate path set is obtained by expanding each path in the preset candidate path set.

[0115] In one optional embodiment, based on the initial probability matrix, the Beam Search algorithm is used to first expand the preset initial candidate path set (including empty paths) to form a target candidate path set containing multiple possible text paths. Then, the recognition reliability of each character on each path is quantified by calculating the entropy value to obtain multiple confidence values.

[0116] For example, the preset candidate path set Beam is a set containing an empty path, with an initial score of 1.0, indicating an initial confidence level of 100%. Then, for each time step t of the initial probability matrix, for each path in Beam, based on the character probability distribution at that time step, all possible characters are expanded, i.e., each character is taken as the next node of the path, thereby forming a new target candidate path set Candidates.

[0117] Furthermore, for each character c_t (corresponding to time step t) on each path in the target candidate path set, the confidence value is calculated using the following relationships (1) and (2): (1) (2) In the formula: Indicates time step The corresponding entropy value is used to quantify the uncertainty of the character recognition result at that position. The larger the entropy value, the higher the uncertainty of the recognition result and the lower the reliability. Represents a single character in the character table; Indicates time step Input text line image block In this case, the location is identified as a character. The probability is derived from the initial probability matrix; Indicates time step Select character The confidence level value after that.

[0118] Step S2033: When multiple confidence values ​​are greater than the preset confidence threshold, the initial candidate path set is updated using the multiple confidence values ​​and the target candidate path set, and multiple target text lines are determined.

[0119] In one optional embodiment, the preset confidence threshold is a reliable threshold determined based on a large amount of handwritten meeting record recognition data. When the confidence of all characters on a path exceeds the threshold, it indicates that the text recognition result corresponding to the path is highly reliable and there are no obvious recognition errors. By updating the initial candidate path set and retaining the optimal path (the path with the highest cumulative score and the most stable confidence), it can be ensured that the final output target text line is the optimal result under the current recognition conditions.

[0120] In an optional embodiment, it is determined whether the confidence values ​​of all characters on each path in the target candidate path set exceed a preset confidence threshold (e.g., ...). If the conditions are met, the initial candidate path set is updated based on the cumulative score and confidence value of the path, the optimal path is retained, and the character sequence corresponding to the optimal path is determined as the target text line for accurate recognition.

[0121] For example, setting a preset reliability threshold Then, for each path in the target candidate path set, check whether the confidence values ​​of all its corresponding characters are greater than 1. Furthermore, retain all characters that satisfy the confidence value for all characters > Candidate paths are selected, and paths containing low-confidence characters are removed.

[0122] Furthermore, the initial candidate path set is updated. Specifically, the selected valid paths are sorted in descending order of cumulative score, retaining the top-ranked paths. Find a path (the same size as Beam) and update the initial candidate path set.

[0123] Furthermore, from the updated initial candidate path set, the path with the highest cumulative score is selected as the optimal path, and the character sequence corresponding to this path is the target text line that is accurately identified.

[0124] Furthermore, the above operation is repeated for the candidate path set corresponding to all target text line image blocks to determine multiple target text lines and associate their corresponding text line bounding box coordinates.

[0125] Step S2034: Generate a target JSON dataset with coordinates based on multiple target text lines, the coordinate information of the multiple target text lines in the target handwritten meeting record image, and multiple confidence values.

[0126] In one optional embodiment, the target JSON dataset is generated by integrating all accurately identified target text lines, the spatial coordinates (bounding boxes) of each target text line in the original target handwritten meeting record image, the character confidence value corresponding to each text line, and other multi-dimensional information, and organizing them in a unified format.

[0127] In some optional implementations, after step S2032, the method further includes: Step b1: When the confidence of the target character in any path in the target candidate path set is less than the preset confidence threshold, extract multiple identified characters from the corresponding target path, and generate target description information based on the bounding boxes of the text lines corresponding to the multiple identified characters and the position of the target character.

[0128] In an optional embodiment, during the Beam Search decoding process, if the confidence level of a target character (corresponding to a specific time step) on any path in the target candidate path set is lower than a preset confidence threshold ( If the identified context characters are extracted from the path, and combined with the bounding box coordinates of the text line and the specific position of the target character in the text line, the target description information containing context, spatial location, and low-confidence character position is formed.

[0129] For example, each path in the target candidate path set is traversed, and the confidence value of each target character on the path is checked one by one to see if it is less than a preset confidence threshold. .

[0130] Furthermore, when a low-confidence target character is detected, the N identified characters preceding the target character and the M identified characters following it are extracted from the current path (i.e., the target path) to form a context character sequence, creating the basic semantic background. Further, the bounding box coordinates corresponding to the text line are retrieved (…). ), and record the time step corresponding to the target character. That is, its position number in the text line.

[0131] Furthermore, information is integrated according to a preset template. An example template is: "Meeting minutes text context: {recognized character sequence}, current text line position coordinates: " The low-confidence character is located at the following position in the text line: "For each character position, please predict the possible characters and candidate words at that position based on the meeting scenario," ultimately generating structured target description information.

[0132] Step b2: Input the target description information into the preset large language model for processing to obtain multiple candidate words and multiple relevance scores of the candidate words.

[0133] In an optional embodiment, the target description information is encapsulated into a Prompt input to a preset Large Language Model (LLM) according to a preset format, and the semantic understanding and contextual reasoning capabilities of the LLM in the meeting scenario are utilized to output multiple candidate words that match the positions of low-confidence characters, as well as the relevance score (0-1 points) of each candidate word with the current context.

[0134] For example, the target description information is first encapsulated in an LLM-compatible format, and the task instruction is added: "Based on the meeting record context and location information, generate 3-5 possible candidate words for this low-confidence location, sort them from high to low relevance, and assign a relevance score between 0 and 1 to each candidate word."

[0135] Secondly, the encapsulated Prompt is input into a pre-defined large language model, and a candidate word list is extracted based on the LLM output (e.g., ...). ) and the corresponding relevance score list (e.g. This ensures that each candidate word is semantically matched to the position of a low-confidence character.

[0136] Step b3: Calculate the normalized score of candidate words containing the target character based on multiple relevance scores.

[0137] In one optional embodiment, based on the relevance scores of multiple candidate words output by LLM, all relevance scores are first normalized, i.e., converted into scores between 0 and 1 with a sum of 1. Then, candidate words containing the target character are selected, their normalized scores are summarized, and finally, the normalized scores of candidate words containing the target character are obtained.

[0138] For example, the normalized score is calculated using the following relations (3) and (4): (3) (4) In the formula: Indicates the first candidate words The normalized score, ranging from 0 to 1, is used to quantify the relative importance of the candidate word among all candidate words. Indicates the relevance score; This indicates all outputs of LLM. The relevance scores of each candidate word are summed. Indicates that it contains the target character The normalized scores of the candidate words.

[0139] Step b4: Weighted fusion of the normalized score and the original probability value of the target character to obtain the target probability value.

[0140] In an alternative embodiment, weighted fusion is performed using the following relation (5): (5) In the formula: Indicates the target probability value; Indicates the fusion weights, for example This indicates a greater emphasis on visual evidence from OCR.

[0141] Step b5: Update the original probability values ​​in the initial probability matrix using the target probability values ​​to obtain the target probability matrix.

[0142] In one optional embodiment, the target probability value of the obtained target character is used to replace the original probability value of the target character at the corresponding time step in the initial probability matrix, while the probability values ​​of the other characters remain unchanged, thus forming the updated target probability matrix.

[0143] Step b6, based on the target probability matrix, returns the steps for calculating multiple confidence values, iterating repeatedly until multiple target text lines are obtained.

[0144] In an optional embodiment, the updated target probability matrix is ​​used as the new calculation basis, and the process returns to step S2033. Steps S2033 to S2034 are repeated until the character confidence on all candidate paths is greater than the preset confidence threshold or the preset number of iterations is reached, and finally multiple accurate target text lines are determined.

[0145] In some optional implementations, step S204 above includes: Step S2041: Parse the target JSON dataset and generate a set of text blocks containing spatial context.

[0146] In one alternative embodiment, the JSON data is parsed to extract the original text and coordinates of each text block. and OCR confidence Based on coordinates Calculate the spatial relationship features of text blocks, including relative position, distance and alignment, and output a set A of text blocks with spatial context.

[0147] For example, the target JSON dataset is read and parsed into an ordered list of blocks, where each element corresponds to a text block object. The list order corresponds to the spatial order of the text in the image (by...). (Coordinates in ascending order) Maintain consistency. Furthermore, extract the id (unique identifier) ​​from each text block object. (Original text) (boundary box coordinates) ), Core fields such as (OCR confidence level).

[0148] Furthermore, for each text block Calculate the coordinates of the center point .

[0149] in , .

[0150] Furthermore, for each Calculate its relationship with all other text blocks ( Euclidean distance of the center point Filter by the smallest distance Each text block is used as a list of neighboring blocks. ;right For each block in the calculation, details of relationships such as relative direction, horizontal offset, vertical distance, and bounding box overlap are calculated.

[0151] Furthermore, comparison and The bounding box coordinates of all blocks in the middle, if of With most neighboring blocks The difference is marked as being within a preset small threshold. Similarly, determine right alignment and center alignment, and generate alignment markers.

[0152] Furthermore, for each create The dictionary contains center point coordinates, alignment markers, and details of its relationship with neighboring blocks. Furthermore, it will... The original information and The enhanced text blocks are then integrated to form an enhanced text block object. Furthermore, all enhanced objects form a collection of text blocks containing spatial context.

[0153] Step S2042: Construct the first Prompt based on the set of text blocks.

[0154] In one optional embodiment, the original text, coordinate information, OCR confidence, and spatial context features (such as alignment and relative position) of each text block in the text block set are structured according to a preset template to generate a first prompt that conforms to the understanding habits of LLM.

[0155] For example, the first Prompt is as follows: "You are a meeting minutes analysis expert. Please clean, correct, semantically annotate, and initially structure the following text. The text blocks are arranged in coordinate order:" [Text block 1]:{originaltext)(coordinates:{bbox), confidence:{conf) [Text block 2]:...: Task: --Cleanup: Correcting typos and filling in missing words.

[0156] --Correction: Correct semantic errors based on context.

[0157] -- Semantic tagging: Identify entities (such as names, times, and topics) and keywords.

[0158] --Preliminary structuring: Estimating the hierarchy (such as headings, subheadings, and body text).

[0159] Output JSON format: {id, clean_text, original_text, hierarchy, entities, keywords, confidence, bbox} Step S2043: Process the Prompt using a preset large language model to obtain a target semantic block JSON array.

[0160] In one optional embodiment, the constructed first Prompt is input into a preset large language model. With the help of its semantic understanding and reasoning capabilities, tasks such as text block cleaning and error correction, entity extraction, and hierarchical prediction are completed. At the same time, the LLM self-evaluates the confidence of task execution. Finally, the processing results are parsed and integrated into a target semantic block JSON array containing multi-dimensional high-quality information.

[0161] For example, the first prompt is input into a preset large language model, reasonable parameters are set, and a processing request is submitted.

[0162] Furthermore, the LLM performs the following operations based on the Prompt instruction: (1) Cleaning and error correction: For text blocks with OCR confidence below the threshold, combine context and spatial relationship to correct typos and fill in missing words to generate clean_text; (2) Entity extraction: Using standard NER logic, identify entities such as names, times, and topics in the text block, extract core keywords, and generate a list of entities and keywords; (3) Hierarchy prediction: Combine text semantics and spatial features to predict the hierarchy of text blocks and generate the hierarchy; (4) Self-assessment confidence level: assess the confidence level between 0 and 1 for the results of each processing task using confLLM to quantify the reliability of the results.

[0163] Furthermore, it receives the structured results output by LLM, verifies the JSON format specification, and extracts fields such as id, clean_text, hierarchy, entities, keywords, and confLLM from each text block.

[0164] Furthermore, combined with OCR confidence level With LLM self-assessment confidence level The overall confidence level is calculated using the following relationship (6): (6) In the formula: and Indicates the weight.

[0165] Furthermore, the original information of the text block (id, original_text, bbox) and the LLM processing results are integrated. The target semantic blocks (hierarchy, entities, keywords) and overall confidence scores are organized into a JSON array of target semantic blocks in a unified format and output.

[0166] In some optional implementations, step S205 above includes: Step S2051: Global information integration is performed on the target semantic block JSON array to obtain a multimodal context information set.

[0167] In one optional embodiment, the target semantic block JSON array is fully parsed to extract multi-dimensional information such as text semantics, visual layout, and correlation quantification. Then, a semantic correlation matrix is ​​generated through text embedding, and spatial relationship features are integrated to finally form a unified multimodal context information set containing text content, coordinate information, semantic correlation, and visual layout.

[0168] For example, parse the target semantic block JSON array and extract the semantic block's... The core fields, such as (cleaned text), bbox (coordinate information), hierarchy (preliminary hierarchy), entities (entities), and keywords (keywords), form a basic information set.

[0169] Furthermore, the Sentence-BERT model is used to encode the clean_text of each semantic block, generating a high-dimensional text embedding vector; based on the vectors, the cosine similarity between all semantic blocks is calculated to construct a semantic association matrix. Here, the matrix elements represent the semantic association strength between two text blocks.

[0170] Furthermore, based on the bounding box coordinates of each semantic block, visual features such as the relative position, spatial distance, and alignment between text blocks are recalculated and added to the basic information set.

[0171] Furthermore, basic information, semantic association matrix, and visual layout features are integrated according to a preset format to form a multimodal contextual information set.

[0172] Step S2052: Input the multimodal context information set into the second Prompt to obtain a standard nested JSON structure dataset.

[0173] In one alternative embodiment, the second Prompt is generated based on content attribution rules based on semantic relevance and visual layout.

[0174] In one optional embodiment, based on semantically driven and visually guided content attribution determination rules, a second Prompt is constructed, which includes multimodal context information, attribution determination logic, and output format requirements. Then, the multimodal context information set is input into a preset large language model, and the LLM infers the hierarchical attribution relationship of each semantic block, and finally outputs a standard nested JSON structure dataset that conforms to the specification.

[0175] For example, the ownership determination rules are as follows: (1) Semantic-driven attribution rules.

[0176] If the semantic relevance between text block B and text block A is greater than the threshold, and B is within the visual influence range of A, then B may belong to A.

[0177] Calculate the semantic association chain: the transmission of association strength from A to B to C.

[0178] (2) The rules of belonging guided by visual layout.

[0179] Indentation hierarchy: A text block indented on the right belongs to the text block that is adjacent to it on the left and is one level above it.

[0180] Spatial proximity principle: Text blocks that are closely adjacent in the vertical direction are more likely to belong to the same logical group.

[0181] Alignment hierarchy principle: Text blocks with the same alignment may belong to the same level.

[0182] (3) Hybrid reasoning strategy: For each candidate parent node P and unassigned node C, calculate the semantic attribution score (based on semantic relevance) and the visual attribution score (based on spatial relationship + alignment). The comprehensive score = semantic score × visual score. If it is greater than the attribution threshold, then C is assigned to P.

[0183] Furthermore, a second Prompt is constructed based on the reasoning basis, rule description, task requirements, and output format.

[0184] Furthermore, the multimodal context information set is input into the second Prompt, and then the LLM performs hybrid reasoning based on the rules and input data to determine the parent node affiliation of each text block one by one.

[0185] Furthermore, the nested JSON structure output by LLM is received, its format requirements are verified, and any unclaimed nodes are checked. If any are missing, the Prompt is added, and LLM is called again to finally obtain a standard nested JSON structure dataset.

[0186] Step S2053: Based on the multimodal context information set and the standard nested JSON structure dataset, perform automatic topic summarization and grouping to obtain an initial tree-structured JSON object.

[0187] In one alternative embodiment, the semantic blocks are grouped by combining the text embedding vectors in the multimodal context information set and the preliminary hierarchical relationship in the standard nested JSON structure. Then, LLM is used to generate a general topic title for each cluster group, and the topic information is integrated into the nested JSON structure to finally obtain an initial tree-structured JSON object.

[0188] For example, semantic blocks corresponding to all leaf nodes are extracted from a standard nested JSON structure dataset, and a cosine distance matrix is ​​calculated based on the text embedding vectors in the multimodal context information set. Furthermore, the Agglomerative Clustering algorithm is applied for hierarchical clustering, and the optimal number of clusters is determined by the silhouette coefficient, dividing the semantic blocks into multiple related content groups.

[0189] Furthermore, for each cluster group, the clean_text and keywords of all semantic blocks within the group are extracted, a topic-generated Prompt is constructed, and the Prompt is input into the LLM to obtain the topic title of each group.

[0190] Furthermore, the topic title is used as the parent node, and the semantic blocks within the clusters are used as child nodes, all integrated into a standard nested JSON structure. Simultaneously, node attributes (such as topic, level, and entity information) are added to form an initial tree-structured JSON object, which contains complete hierarchical relationships and topic categories.

[0191] Step S2054: Optimize the structure and resolve conflicts of the initial tree-structured JSON object to obtain the target tree-structured JSON object.

[0192] In one optional embodiment, the initial tree-structured JSON object is subjected to logical conflict detection, unreasonable structure correction and sorting optimization to eliminate problems such as circular structure, hierarchical jumps, and multiple parent nodes. The node order is adjusted to fit the spatial layout of the original meeting minutes, and finally a standardized, reasonable and logically rigorous target tree-structured JSON object is generated.

[0193] For example, a graph traversal algorithm is used to detect circular structures in the initial tree structure (e.g., A is the parent node of B, and B is the parent node of A). Further, depth difference checks are used to determine the rationality of the hierarchy and identify hierarchical jumps (e.g., directly from the root node to a third-level child node). Further, it is checked whether the same child node has multiple parent nodes, and all conflict locations and types are recorded.

[0194] Furthermore, for circular structures, the edges with the lowest semantic relevance are deleted. Furthermore, for conflicts between multiple parent nodes, the parent node with the highest semantic relevance is retained. Furthermore, for hierarchical jumps, intermediate topic nodes are inserted to complete the hierarchy.

[0195] Furthermore, the order of child nodes under the same parent node is adjusted, arranged in ascending order according to the y_min coordinate of the original text block (to match the spatial layout of the meeting minutes). Further, node naming and attribute descriptions are optimized to ensure information integrity, ultimately generating a standardized, logically rigorous target tree-structured JSON object.

[0196] In some optional implementations, step S206 above includes: Step S2061: Render the target tree-structured JSON object to obtain the initial mind map component.

[0197] In one alternative embodiment, based on the target tree-structured JSON object, the abstract structured data can be transformed into a graphical component that conforms to the visual specifications of mind mapping, i.e., the initial mind map component, using a visualization rendering tool (such as Echarts).

[0198] For example, the target tree-structured JSON object is read, and the core fields of each node, such as node_id, parent_id, content (topic / key points), children (list of child nodes), entities (entities), and keywords (keywords), are extracted. Then, the JSON data format is converted into an input format supported by the visualization tool, and a list of node data and a hierarchical relationship mapping table are constructed.

[0199] Furthermore, configuring the visual rules and layout parameters of the mind map can include: (1) Layout method: adopt a radial tree layout or a horizontal / vertical tree layout, with the root node located in the center or on the left, and the child nodes arranged outward / to the right in order of hierarchy; (2) Style configuration: The root node uses a prominent style (such as large font and dark background), and different levels of nodes use different font sizes and colors (such as red for the title level and black for the body level). The node borders and spacing are set according to visual comfort. (3) Tag display: The node tag displays the core content, and when the mouse hovers over it, it displays supplementary information such as entities and keywords.

[0200] Furthermore, the tool calls Echarts' mind map rendering API, passing in the adapted node data and configuration parameters to trigger the rendering process. Then, it automatically calculates the coordinates of each node based on the hierarchical relationship, draws the node graphics and hierarchical connection lines, and generates an initial mind map component with a complete logical layout and visual style.

[0201] Step S2062: Add a click event to the initial mind map component to obtain the target mind map component.

[0202] In one optional embodiment, front-end interaction technology is used to bind click events (such as expand / collapse) to each node in the initial mind map component, so as to realize the dynamic display and hiding of the node hierarchy, thereby obtaining a target mind map component with interactive capabilities.

[0203] For example, all node DOM elements in the initial mind map component are traversed, and a click event listener is bound to each node; the event triggering objects include the node text label and the node graphic area, ensuring that the user can trigger the interaction by clicking any position of the node.

[0204] Furthermore, write the click event handling function and define the core logic: (1) Status judgment: Get the expanded status flag of the current node (such as the isExpanded attribute). If it is false (not expanded), read the child node data of the node from the target tree structure JSON object, call the visualization tool API to render the child node graphics, add it to the lower level area of ​​the current node, update isExpanded to true, and switch the icon next to the node to "-". (2) Reverse operation: If the node status is true (expanded), remove the DOM elements of all child nodes under the node, update isExpanded to false, and toggle the icon to "+"; (3) Adaptive layout: When child nodes are shown / hidden, the position of the parent node and nodes at the same level are automatically adjusted to ensure that the overall layout is compact and without overlap.

[0205] Furthermore, after completing the event binding and interaction logic configuration of all nodes, a target mind map component with expand / collapse interaction function is obtained, which allows users to control the display status of node levels as needed.

[0206] Step S2063: Transform the elements in the target mind map component to obtain the target mind map.

[0207] In an alternative embodiment, a graphics conversion tool (such as a dom-to-image library) can be used to convert the interactive target mind map component (DOM element) into a common image format (PNG or SVG) and allow users to export and save it.

[0208] For example, first, import the dom-to-image library and configure the conversion parameters, which may include: (1) Image format: Supports user selection of PNG (default) or SVG format; (2) Resolution and quality: PNG format is set to 300 DPI to ensure image clarity; SVG format retains the original vector information without loss of image quality; (3) Scope specification: Clearly define the conversion scope as the root DOM element of the target mind map component, and avoid including other irrelevant elements of the webpage.

[0209] Furthermore, before performing the conversion, the current interactive state of the target mind map components is fixed (such as the nodes that the user has expanded and the current view position) to ensure that the converted image accurately presents the content that the user is currently viewing.

[0210] Furthermore, the conversion API of the dom-to-image library (such as toPng() or toSvg()) is called, passing in the root DOM element of the target mind map component and configuration parameters to start the conversion process. The tool then parses the structure, style, and content of the DOM element and generates the corresponding image data.

[0211] Furthermore, the converted image data is encapsulated into a downloadable file object, providing users with an export entry point (such as a download button). After the user clicks download, the image file (PNG / SVG format) is saved locally, ultimately resulting in a target mind map that can be stored, shared, and printed.

[0212] In one example, a method and system for automatically generating interactive mind maps from handwritten meeting minutes based on a multimodal fusion large language model is provided. This system constructs an end-to-end artificial intelligence system with LLM as the intelligent hub, utilizing a dual error correction and enhancement mechanism of "domain-adaptive OCR" and "LLM semantic understanding," rather than simply stringing together independent tools. The main steps include: (1) Multimodal image preprocessing and domain-adaptive OCR recognition module.

[0213] This module designs an architecture that allows OCR and LLM to work together deeply. OCR and LLM are no longer independent but rather a dual-engine system that assists and verifies each other. The core task is to convert unstructured, potentially noisy, handwritten meeting transcripts into high-accuracy, structured text. The flowchart is as follows: Figure 3 As shown.

[0214] This module is divided into two phases. Phase one improves image quality through multimodal image preprocessing, creating an ideal input environment for OCR recognition and minimizing noise and distortion introduced during physical acquisition. Phase two involves domain-adaptive OCR and real-time semantic collaboration, which is a key manifestation of dual-engine synergy.

[0215] ① Multimodal image preprocessing stage. A corpus containing a large amount of handwritten meeting record data from various sources, including whiteboard photos of handwritten meeting minutes, scanned copies of paper notes, and input from electronic handwriting tablets, is used to perform domain-specific fine-tuning on the pre-trained OCR model, adapting it to the "handwritten meeting style." Specific steps are as follows: 1) Image acquisition and modal alignment. Feature point matching is used to stitch multiple images together to ensure complete coverage of the text region. Image registration is achieved through the Homography matrix H.

[0216] 2) Image enhancement. A nonlocal means denoising algorithm is applied to denoise the image; an adaptive thresholding method is used to convert the image into a black-and-white binary image to highlight text areas.

[0217] 3) Geometric correction. Detect text tilt angle. And perform rotational correction.

[0218] 4) Text Region Detection and Cropping. A deep learning-based text detection model is used to locate text regions and crop out the ROIs to reduce the scope of subsequent OCR processing.

[0219] ② Domain-Adaptive OCR and Real-Time Semantic Collaboration Stage. During the decoding stage of OCR recognition, when the model's confidence level in recognizing a certain character or word falls below a set threshold, the LLM engine is invoked in real time. The LLM provides a list of the most likely candidate words based on the recognized contextual semantics (such as the meeting topic, the preceding content), guiding the OCR model to make judgments and selections, greatly improving recognition accuracy in complex scenarios. The specific steps are as follows: 1) Text detection: bounding box extraction and location information processing.

[0220] The preprocessed meeting transcript image is input into the fine-tuned text detection model, which outputs bounding boxes for all lines of text. Each bounding box is represented as... , where the coordinates are the pixel positions in the original image. Based on each The corresponding text line image block I is cropped from the original image for use by the recognition module. All Store the coordinate information in a list ,in This represents the total number of lines of text detected. This information will serve as a crucial visual layout clue and will be passed to subsequent LLM processing modules.

[0221] 2) Text recognition: from image to character probability sequence.

[0222] The cropped text line image Scale to a fixed size. Image I is passed through a CNN backbone network, outputting a feature map. The feature map is then pooled along the height dimension and expanded along the width dimension to obtain the feature sequence. ,in For sequence length, ( (For feature dimensions). The feature sequence Input a Transformer encoder to perform global context modeling. Output the enhanced feature sequence. ,in For each time step Features By using a fully connected layer and a softmax function, the probability distribution of all characters at that position is obtained: .

[0223] The final output is a probability matrix. , of which elements Indicates at time step ,image The corresponding position is a character probability .

[0224] 3) Decoding: From probability sequence to text string and confidence calculation.

[0225] Maintain a size of The candidate path set Beam is initially set to contain one empty path with a score of 1. For each path in Beam, all possible next characters are expanded to form a new candidate path set Candidates. The score of each new path is calculated. For each path at time step... Selected character Its confidence level Calculated using the above relations (1) and (2).

[0226] Select the candidate with the highest score. The path is identified and the Beam is updated. After processing all time steps, the path with the highest score in the Beam is the final identified text line.

[0227] 4) LLM reverse guidance: real-time semantic collaborative error correction.

[0228] During the Beam Search decoding phase, when a path is at time step Extended characters At that time, if its confidence level We introduce the semantic knowledge of LLM for real-time guidance and correct the decoding path.

[0229] When the confidence level is less than the threshold, encapsulate the request for help information and extract the first N recognized characters or words from the current path. Based on the text line... Based on the approximate position of the current character, generate a description and record the positions of low-confidence characters. The encapsulated information is sent to the LLM via Prompt.

[0230] The Prompt is as follows: In the meeting minutes, the context is C, and the current position is within the document. For the position, please predict the 3-5 most likely candidate companies to appear next, please sort them in descending order of probability, and give a confidence score between 0 and 1.

[0231] LLM returns a structured list of candidates. ,in, Indicates LLM for candidate words The relevance score for the character. Calculate its LLM probability for all included The sum of the normalized scores of the candidate words is shown in the above relations (3) and (4).

[0232] Furthermore, the weighted average is used to fuse the original OCR probability and the LLM probability, as shown in the above relation (5).

[0233] Furthermore, using the new probability distribution after fusion Replace the original At time step The distribution of BeamSearch is used to recalculate all candidate paths at time step based on the updated probabilities. The score is calculated, and the path selection step is re-executed.

[0234] The decoding process continues until text line recognition is complete. The final output of this module is a structured JSON file containing location and confidence information, including the content of each recognized text segment, its coordinates in the original image, and its confidence score.

[0235] (2) LLM-driven semantic understanding and structured cleaning module.

[0236] This module receives the JSON text data with coordinates and confidence scores output from step one. Its core task is to use a Large Language Model (LLM) as the central processor to perform deep cleaning, error correction, semantic annotation, and preliminary structuring of the text from the previous stage, preparing high-quality "semantic blocks" for subsequent deep parsing. The flowchart is as follows: Figure 4 As shown.

[0237] ① Input data parsing and spatial relationship calculation.

[0238] Parse JSON data and extract the original text and coordinates of each text block. and OCR confidence Based on coordinates The program calculates the spatial relationship features of text blocks, including relative position, distance, and alignment, and outputs a set A of text blocks with spatial context. Specific steps include: 1) Data Parsing: Read the JSON file and parse it into an ordered list of blocks, where each element corresponds to a text block object; ensure that the list order matches the spatial order of the text in the image; this order is typically determined by the OCR module based on the bounding boxes. (or Output the coordinates after sorting.

[0239] 2) Geometric attribute calculation: For each text block in the blocks list Extract fields and calculate center point Used for subsequent distance and direction calculations.

[0240] 3) Spatial Relationship Analysis: To reduce computational load and focus on meaningful local relationships, only for each Calculate its with The nearest neighbor spatial relationship. For all other text blocks. ( ) calculation and Euclidean distance of the center point Choose the one with the smallest distance. Each block as Neighboring block list ,right Each neighboring block in Calculate direction, horizontal offset, vertical distance, and bounding box overlap. For all other text blocks... ( ), calculation and Euclidean distance of the center point Choose the one with the smallest distance. Each block as Neighboring block table .

[0241] 4) Alignment determination: Compare bounding box coordinates to determine... and The overall alignment style of all adjacent blocks in the middle. If of With most neighboring blocks If the difference is within a small threshold, it is marked as... (Left-aligned); Comparison Mark right-aligned; compare The marker is centered.

[0242] 5) Feature integration: for each Create A dictionary containing: "centroid" (center point coordinates); "alignment" (alignment marker); and "relative_positions" (a list containing details of its relationship with neighboring blocks, each detail being a dictionary containing...). , direction , =and other fields. Each All the information is integrated into a new dictionary and placed into a set A, which is a list where each element is an enhanced text block object that contains both the raw data provided by OCR and the calculated, semantically rich spatial relationship features.

[0243] ② Create an LLM Prompt. Construct a Prompt from the data A output in the previous step according to a preset template, and send it to the LLM. Output data B. When creating the Prompt, not only is the text provided, but the spatial relationships between text blocks are also provided to the LLM as key contextual information, guiding the LLM to incorporate visual layout cues from the early stages of semantic understanding. The format is as follows: "You are a meeting minutes analysis expert. Please clean, correct, semantically annotate, and initially structure the following text. The text blocks are arranged in coordinate order:" [Text block 1]:{originaltext)(coordinates:{bbox), confidence:{conf) [Text block 2]:...: Task: --Cleanup: Correcting typos and filling in missing words.

[0244] --Correction: Correct semantic errors based on context.

[0245] -- Semantic tagging: Identify entities (such as names, times, and topics) and keywords.

[0246] --Preliminary structuring: Estimating the hierarchy (such as headings, subheadings, and body text).

[0247] Output JSON format: {id, cleantext, originaltext, hierarchy, entities, keywords, confidence, bbox} ③LLM processing and calculation.

[0248] 1) Cleaning and Error Correction: LLM corrects OCR errors based on a semantic understanding and context (including spatial relationships) of the entire document. The prompt will specifically instruct LLM to focus on those OCR confidence levels (…). (Below the threshold) The text blocks. Output data C, which is the clean_text field of each text block.

[0249] 2) Entity Extraction: LLM extracts data from C (… Based on this, predefined entities are identified and core keywords are extracted. Entity recognition uses the standard NER method, but is implemented by an LLM, requiring no additional model. For each text block, the LLM outputs D data, namely, a list of entities and a list of keywords for each text block.

[0250] 3) Hierarchy prediction: LLM predicts the hierarchy data (E) based on the semantic content of the text and the coordinate space relationship.

[0251] 4) LM self-assessment of confidence: For each task it completes (such as error correction, entity recognition), LLM will self-assess a confidence score. This is achieved by requesting the LLM to output a value between 0 and 1 in the Prompt. The output is F data, which represents the specific task-related parameters for each text block. .

[0252] 5) Calculate the overall confidence level: Combine the OCR confidence level (from data A) and the LLM processing confidence level (from data F) to update the final confidence level, as shown in the above relation (6).

[0253] ④ Enhance structured JSON output: Extract the id from data A, (Original text), bbox (coordinates); in C data (Final text); entities (entities list) and keywords (keywords) from the D data; hierarchy (estimated hierarchy) from the E data; confidence (overall confidence level) from the G data. Input is into a unified, structured JSON object. Output is a final high-quality, enhanced semantic block JSON array.

[0254] (3) Deep semantic analysis and structured generation module of meeting content.

[0255] The core of this module lies in integrating visual layout information and textual semantic information for joint reasoning to generate a global logical structure. The text position coordinates (visual layout information) provided by OCR, the preliminary hierarchical predictions and entity information provided by LLM, and the global semantic association matrix together form a rich context. This context is input into LLM, which analyzes the semantic relationships between all blocks, infers their hierarchical relationships, summarizes and groups them, and finally generates a complete, tree-like structure—the blueprint for the mind map. This module consists of four sub-steps: ① Global information integration.

[0256] The precise text location coordinates (visual layout information) provided by OCR, the preliminary hierarchical prediction and entity information provided by LLM, and the global semantic association matrix generated by Sentence-BERT together constitute a rich multimodal context. The text content, coordinate information, and preliminary hierarchy of all semantic blocks are extracted from the output of the previous module, and Sentence-BERT is used to analyze each semantic block. Text embedding vectors are generated, and a semantic association matrix is ​​calculated based on these vectors. Simultaneously, visual layout relationships (relative position, alignment, and spatial distance) are calculated based on bounding box coordinates. This integrates semantic information, visual layout information, and the semantic association matrix into a unified multimodal context. This step integrates information from different sources and types (textual semantics, visual layout, and numerical association) into a unified, machine-understandable data structure, denoted as data F.

[0257] ②LLM-driven logical structure reasoning.

[0258] This step involves determining content ownership based on semantic relevance and visual layout. The ownership determination rules are as follows: (1) Semantic-driven attribution rules.

[0259] If the semantic relevance between text block B and text block A is greater than the threshold, and B is within the visual influence range of A, then B may belong to A.

[0260] Calculate the semantic association chain: the transmission of association strength from A to B to C.

[0261] (2) The rules of belonging guided by visual layout.

[0262] Indentation hierarchy: A text block indented on the right belongs to the text block that is adjacent to it on the left and is one level above it.

[0263] Spatial proximity principle: Text blocks that are closely adjacent in the vertical direction are more likely to belong to the same logical group.

[0264] Alignment hierarchy principle: Text blocks with the same alignment may belong to the same level.

[0265] (3) Hybrid reasoning strategy: FOR each candidate node P: FOR each unassigned node C: Calculate the semantic attribution score = f(semantic relevance score, P, C). Calculate the visual attribution score = g(spatial relationship (P,C), alignment (P,C)). Overall Attribution Score = Semantic Attribution Score × Visual Attribution Score If the overall attribution score is greater than the attribution threshold, then the attribution score is greater than the attribution threshold. Assign C to be a child node of P. The above rules are constructed as an LLM Prompt. The data F is input, and the LLM is required to output a standard nested JSON structure, which is the data G. The LLM is also required to briefly explain its reasoning process before outputting the final result, which helps to improve the credibility and accuracy of the structure generation.

[0266] The LLM Prompt is as follows: Please perform logical structure reasoning based on the following dimensions of information: 1. Semantic association matrix (numerical representation of association strength): [Display semantic association matrix] 2. Visual layout relationship - Text block A is located below text block B and has the same left alignment. - Text block C is to the right of text block D, and is relatively close. -[Display spatial relationship description] 3. Attribution determination task: Please determine the parent node for each text block, based on the following criteria: a) Semantic association strength with candidate parent nodes b) Spatial relationship in the original layout C) Logical relevance of text content Please output the reasoning process first, and then output the standard ISON structure.

[0267] ③ Automatic topic summarization and grouping.

[0268] The input for this step is data G (the initial tree structure generated by LLM) and text embedding vectors from data F. The text content of all leaf nodes is extracted from data G. A hierarchical clustering algorithm is used to group the data based on the text embedding vectors. Specifically, the steps include calculating the cosine distance matrix, applying Agglomerative Clustering for clustering, determining the optimal number of clusters based on the silhouette coefficient, and finally, generating a summary topic title for the text content in each cluster using LLM. This clustering topic information is then integrated into the tree structure. The final output is data H. The Prompt command is as follows: For the following set of relevant meeting content: {List of text contents in clusters} Please generate a concise title (no more than 10 characters) that accurately summarizes the common theme of this group of content.

[0269] Requirements: The minutes should be concise, clear, and in accordance with meeting minutes style.

[0270] ④ Structural optimization and conflict resolution.

[0271] The tree structure generated by LLM undergoes final verification and optimization to handle potential logical conflicts. Conflict detection includes using graph traversal algorithms to detect circular structures, using depth difference checks to verify the rationality of hierarchical levels, and checking the relevance of child nodes under the same parent node. Optimization includes resolving conflicts between multiple parent nodes (selecting the node with the highest semantic relevance), correcting unreasonable hierarchical jumps (inserting intermediate nodes or adjusting hierarchies), and final sorting adjustments based on visual layout (sorting by the original Y-coordinate).

[0272] The output of this module is a complete, standardized tree-structured JSON object. This object precisely describes the content, attributes, and hierarchical relationships between all nodes in the mind map, serving as a direct blueprint for generating the mind map.

[0273] (4) Interactive mind map automatic generation and output module.

[0274] This module receives the tree-structured data generated in step three. Its core task is to render this data into a visual, interactive mind map component using Echarts, and simultaneously bind click events (expand / collapse) to nodes using the API provided by Echarts. It uses the dom-to-image library to convert the mind map DOM elements in the webpage into PNG or SVG images and supports export. Finally, it encapsulates the mind map component into an independent service or component that can be called by other systems.

[0275] This example provides a method and system for automatically generating interactive mind maps from handwritten meeting minutes based on a multimodal fusion large language model, which has the following effects: (1) It greatly improves the efficiency and automation of meeting information processing, overturns the inefficient process of traditional “handwritten notes → manual typing → logical organization → manual drawing of mind maps”, integrates multiple completely separate processing steps into an organic intelligent system, and realizes end-to-end intelligent automation.

[0276] (2) Multimodal fusion improves processing accuracy: Through the dual-engine collaborative architecture of "visual perception-semantic cognition", real-time, bidirectional information interaction between OCR and LLM is realized. LLM can use its powerful semantic understanding capabilities to provide real-time error correction suggestions for low-confidence OCR recognition results, thereby significantly improving the first-pass recognition accuracy of handwritten text, especially illegible handwriting and professional terminology. At the same time, the confidence information output by OCR can guide LLM to carefully process uncertain text. This collaborative mechanism fundamentally blocks the error propagation path and greatly enhances the robustness and reliability of the entire system when processing complex real-world data.

[0277] (3) It achieves deep integration of visual layout and deep semantics, accurately restoring the logical structure of the document. The visual layout information of the text is elevated to a core input modality of equal importance to the text content. By deeply integrating and jointly reasoning with the semantic understanding capabilities of the large language model, the system can automatically identify and reconstruct the hierarchical relationships between topics, key points, and sub-items in meeting minutes with extremely high accuracy, thereby generating a logically clear and structurally rigorous mind map.

[0278] (4) The output is intuitive and valuable: Based on the logical relationship diagram, a visual layout is created to generate a mind map with a clear structure and highlighting key points, which directly provides high-quality materials for meeting review, task tracking and knowledge accumulation.

[0279] This embodiment also provides an apparatus for automatically generating mind maps of handwritten meeting minutes. This apparatus is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0280] This embodiment provides a device for automatically generating mind maps of handwritten meeting minutes, such as... Figure 5 As shown, the device includes: The acquisition module 501 is used to acquire multiple original handwritten meeting record images of the target meeting.

[0281] The first processing module 502 is used to perform multimodal preprocessing on multiple original handwritten meeting record images to obtain the target handwritten meeting record image.

[0282] The second processing module 503 is used to generate a target JSON dataset with coordinates based on the target handwritten meeting record image, through target OCR model recognition and decoding and preset large language model collaborative processing. The target OCR model and preset large language model work together and are integrated in the dual-engine module.

[0283] The third processing module 504 is used to perform semantic understanding and processing based on the target JSON dataset using a preset large language model, and obtain a target semantic block JSON array.

[0284] Parsing module 505 is used to perform deep semantic parsing of the meeting content of the target meeting using the target semantic block JSON array, and generate a target tree structure JSON object.

[0285] Module 506 is used to generate a target mind map of the target meeting using the target tree structure JSON object.

[0286] The apparatus for automatically generating mind maps of handwritten meeting minutes provided in this embodiment of the invention can execute the method for automatically generating mind maps of handwritten meeting minutes provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the above modules are the same as in the corresponding embodiments described above, and will not be repeated here.

[0287] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0288] The following is a detailed reference. Figure 6 This diagram illustrates a suitable structural design for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0289] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0290] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the method for automatically generating mind maps of handwritten meeting minutes according to embodiments of the present invention.

[0291] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0292] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the method for automatically generating mind maps of handwritten meeting minutes shown in the above embodiments is implemented.

[0293] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0294] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for automatically generating mind maps of handwritten meeting minutes, characterized in that, The method includes: Acquire multiple original handwritten meeting transcript images of the target meeting; Multimodal preprocessing is performed on the multiple original handwritten meeting record images to obtain the target handwritten meeting record image; Based on the target handwritten meeting record image, after recognition and decoding by the target OCR model and collaborative processing by the preset large language model, a target JSON dataset with coordinates is generated. The target OCR model and the preset large language model run collaboratively and are integrated in the dual-engine module. Based on the target JSON dataset, semantic understanding and processing are performed using the preset large language model to obtain a target semantic block JSON array; Using the target semantic block JSON array, deep semantic parsing is performed on the meeting content of the target meeting, and a target tree-structured JSON object is generated; Using the target tree-structured JSON object, generate a target mind map for the target meeting.

2. The method according to claim 1, characterized in that, Multimodal preprocessing is performed on the multiple original handwritten meeting record images to obtain the target handwritten meeting record image, including: The multiple original handwritten meeting record images are stitched together and registered to obtain the first handwritten meeting record image; The first handwritten meeting record image is enhanced to obtain the second handwritten meeting record image; Geometric correction is performed on the second handwritten meeting record image to obtain the third handwritten meeting record image; The text region of the third handwritten meeting record image is located and cropped using a text detection model to obtain the target handwritten meeting record image.

3. The method according to claim 1, characterized in that, Based on the target handwritten meeting record image, after target OCR model recognition and decoding and pre-set large language model collaborative processing, a target JSON dataset with coordinates is generated, including: Based on the target handwritten meeting record image, the target OCR model is used to perform text detection and recognition to obtain an initial probability matrix of multiple characters; Based on the initial probability matrix, multiple confidence values ​​of multiple characters in each path of the target candidate path set are calculated, wherein the target candidate path set is obtained by expanding each path in the preset candidate path set; When all of the multiple confidence values ​​are greater than a preset confidence threshold, the initial candidate path set is updated using the multiple confidence values ​​and the target candidate path set, and multiple target text lines are determined; Based on the multiple target text lines, the coordinate information of the multiple target text lines in the target handwritten meeting record image, and the multiple confidence values, generate the target JSON dataset with coordinates.

4. The method according to claim 3, characterized in that, Based on the target handwritten meeting transcript image, the target OCR model is used for text detection and recognition to obtain an initial probability matrix of multiple characters, including: The target handwritten meeting record image is input into the target OCR model to obtain multiple text line bounding boxes; Based on the multiple text line bounding boxes, multiple initial text line image blocks are determined in the target handwritten meeting record image; The multiple initial text line image blocks are scaled to obtain multiple target text line image blocks; The multiple target text line image blocks are input into the CNN backbone network for processing, and an initial feature sequence is determined; The initial feature sequence is input into the Transformer encoder for processing to obtain the target feature sequence; The target feature sequence is processed through a fully connected layer and a Softmax function to generate the initial probability matrix.

5. The method according to claim 3, characterized in that, The method further includes: When the confidence of a target character in any path in the target candidate path set is less than the preset confidence threshold, multiple identified characters are extracted from the corresponding target path, and target description information is generated based on the bounding box of the text line corresponding to the multiple identified characters and the position of the target character. The target description information is input into the preset large language model for processing to obtain multiple candidate words and multiple relevance scores of the multiple candidate words; Based on the multiple relevance scores, the normalized scores of candidate words containing the target character are calculated; The normalized score and the original probability value of the target character are weighted and fused to obtain the target probability value; The original probability values ​​in the initial probability matrix are updated using the target probability values ​​to obtain the target probability matrix; Based on the target probability matrix, the steps for calculating the multiple confidence values ​​are returned, and this process is repeated iteratively until the multiple target text lines are obtained.

6. The method according to claim 1, characterized in that, Based on the target JSON dataset, semantic understanding and processing are performed using the preset large language model to obtain a target semantic block JSON array, including: The target JSON dataset is parsed to generate a set of text blocks containing spatial context; Construct a first Prompt based on the set of text blocks; The Prompt is processed by the preset large language model to obtain the target semantic block JSON array.

7. The method according to claim 1, characterized in that, Using the target semantic block JSON array, deep semantic parsing is performed on the meeting content of the target meeting, and a target tree-structured JSON object is generated, including: Global information integration is performed on the target semantic block JSON array to obtain a multimodal context information set; The multimodal context information set is input into the second Prompt to obtain a standard nested JSON structure dataset, wherein the second Prompt is generated according to the content ownership determination rules based on semantic relevance and visual layout; Based on the multimodal context information set and the standard nested JSON structure dataset, topics are automatically summarized and grouped to obtain an initial tree-structured JSON object; The initial tree-structured JSON object is optimized and conflict resolved to obtain the target tree-structured JSON object.

8. The method according to claim 1, characterized in that, Using the target tree-structured JSON object, generate a target mind map for the target meeting, including: Render the target tree-structured JSON object to obtain the initial mind map component; Add a click event to the initial mind map component to obtain the target mind map component; The elements in the target mind map component are transformed to obtain the target mind map.

9. A device for automatically generating mind maps of handwritten meeting minutes, characterized in that, The device includes: The acquisition module is used to acquire multiple original handwritten meeting record images of the target meeting; The first processing module is used to perform multimodal preprocessing on the multiple original handwritten meeting record images to obtain the target handwritten meeting record image; The second processing module is used to generate a target JSON dataset with coordinates based on the target handwritten meeting record image, through target OCR model recognition and decoding and preset large language model collaborative processing. The target OCR model and the preset large language model run collaboratively and are integrated in the dual-engine module. The third processing module is used to perform semantic understanding and processing based on the target JSON dataset using the preset large language model to obtain a target semantic block JSON array; The parsing module is used to perform deep semantic parsing on the meeting content of the target meeting using the target semantic block JSON array, and generate a target tree structure JSON object; The generation module is used to generate a target mind map of the target meeting using the target tree-structured JSON object.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to execute a method for automatically generating mind maps of handwritten meeting minutes as described in any one of claims 1 to 8.

Images