A meeting minutes automatic generation method and system based on OCR technology

Abstract

The application discloses a conference minutes automatic generation method and system based on an OCR technology. Main technical means: multi-modal data is collected through a multi-view camera and a microphone array, a PaddleOCR, speech recognition and a CNN-LSTM network are used to detect gestures, data is fused through a multi-modal time synchronization and a dynamic weight distribution model, structured output is carried out according to a self-defined semantic template, and the system is deployed on an edge computing device, and the system has functions of real-time error correction, field self-adaption, intelligent distribution and the like. The system can realize comprehensive and accurate collection and real-time processing of conference information, improve efficiency and standardization of minutes generation, enhance adaptability and reliability of the system, and realize closed-loop automatic management of conference information.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method and system for automatically generating meeting minutes based on OCR technology. Background Technology

[0002] Traditional meeting minutes generation relies on manual recording, which is inefficient and prone to missing key information. Existing automatic generation technologies are mostly based on a single modality, such as speech recognition, and cannot fully capture visual information in a meeting, such as whiteboard content, projected documents, and non-verbal signals, such as speaker gestures. For example, speech recognition alone may lead to transcription errors due to accents and background noise, while static text analysis struggles to recognize dynamically emphasized content. Furthermore, existing technologies lack the ability to fuse multimodal data in real time and output structured data, failing to meet the needs of efficient meeting management. Summary of the Invention

[0003] The purpose of this invention is to overcome one or more shortcomings of the prior art and provide a method and system for automatically generating meeting minutes based on OCR technology. By integrating PaddleOCR, speech recognition and gesture detection, it can achieve real-time capture of meeting content, multimodal data fusion and structured output, significantly improving the accuracy and generation efficiency of meeting minutes.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A method for automatically generating meeting minutes based on OCR technology includes the following steps:

[0006] By deploying multi-view cameras and microphone arrays in the venue, visual image streams and surround audio streams containing whiteboard content, projected images and speaker gestures are captured simultaneously.

[0007] The visual image stream is sequentially subjected to PaddleOCR text detection, recognition, and layout analysis to obtain structured text data; the surround audio stream is subjected to end-to-end speech recognition and entity relation extraction to generate time-stamped speech text; the gesture action detection step: the speaker's gestures are detected in real time using a CNN-LSTM network model, and the gesture action type and time node corresponding to key content are marked;

[0008] Multimodal data processing: Visual text, speech text, and gesture-tagged data are aligned based on a multimodal time synchronization algorithm. An attention mechanism is used to construct a dynamic weight allocation model, denoted as [model name missing]. Attention weights The text embedding vector for OCR recognition. Standard speech-to-text embedding vector; For the embedding dimension, the corresponding formula is: ,in It is a weight matrix. It's a bias term, a sigmoid function, and it automatically adjusts the fusion priority of OCR text and speech text for the corresponding time period based on the gesture type. It is the fused text representation, calculated using the following formula: ;

[0009] Following a predefined structured semantic template, the fused multimodal data is automatically populated into the structured fields of the topic categories, decision items, and action item lists, generating meeting minutes documents with key annotations.

[0010] Furthermore, in the gesture detection step, the CNN-LSTM network model is trained in the following manner:

[0011] We used the publicly available Jester gesture dataset for pre-training to extract spatiotemporal features of gestures;

[0012] By leveraging custom gesture data from meeting scenarios for transfer learning, the parameters of the action classifier are optimized and updated. ,in Is In the next iteration, the first-order matrix of the gradient is estimated. It is a second-order estimate of the gradient. For learning rate, It is a small constant;

[0013] Custom gesture data includes: pointing, selection, erasing, and zooming gestures;

[0014] The output includes a confidence score for the gesture and the coordinates of the corresponding image region.

[0015] Furthermore, the multimodal time synchronization algorithm is specifically as follows:

[0016] The hardware clock calibration module for the camera and microphone controls the timestamp error between the visual and audio frames within the set range.

[0017] Based on the dynamic time warping algorithm, non-linear alignment is performed on the OCR text generation time, speech recognition time, and gesture detection time.

[0018] Furthermore, the method for constructing the dynamic weight allocation model includes:

[0019] When a "circle" or "point" gesture is detected, the weight coefficient of the corresponding OCR text area is increased to the set multiple of the voice text.

[0020] When a "wipe" gesture is detected, the filtering mechanism for the corresponding time period is triggered;

[0021] The weighting coefficients are dynamically adjusted using a reinforcement learning model trained on historical conference data.

[0022] Furthermore, the structured semantic template can be customized by the user, including:

[0023] Configurable field mapping rules: Automatically map entity types in multimodal data to template fields;

[0024] Adjustable output formatting rules: Generate rich text documents with a table of contents, highlighted text, and associated attachments.

[0025] Furthermore, it also includes a real-time error correction mechanism:

[0026] Contextual semantic verification is performed on the OCR recognition results, and text coherence is detected using the BERT model, assuming a sentence... Take the CLS token vector output by BERT. As a sentence vector, the formula for calculating cosine similarity is: For recognition results with a confidence level lower than the set value, a secondary recognition is triggered. The calculation formula is as follows: ,in To adjust the hyperparameters of the slope;

[0027] in, For the OCR-recognized sentence to be verified, For contextual reference sentences, For sentences The semantic vector (the CLS token sentence vector output by the BERT model). For sentences semantic vectors (and) (Sentence vector of the same dimension as the context reference sentence).

[0028] Speaker separation processing is performed on the speech recognition results, and cross-speaker transcription errors are corrected by combining the conference room seating layout information.

[0029] Furthermore, the multimodal data processing is deployed on an edge computing device, specifically as follows:

[0030] NVIDIA Jetson AGX Orin is used as the edge node to achieve local real-time inference and end-to-end latency processing;

[0031] The raw data is encrypted and transmitted to the cloud via the network for model updates and historical data archiving.

[0032] Furthermore, the PaddleOCR processing module includes a trainable domain-adaptive model:

[0033] For specific professional fields, the text detection and recognition branches of PP-OCRv4 were fine-tuned using a small amount of labeled data;

[0034] Perform end-to-end recognition of mixed handwritten and printed text scenarios.

[0035] Furthermore, after the meeting minutes document is generated, an intelligent distribution step is also included:

[0036] By using NLP technology to extract key participant information from documents, to automatically generate to-do reminders and push them to the enterprise collaboration platform;

[0037] Generate an audio summary file with timestamp index and associate it with the corresponding paragraph of the meeting minutes.

[0038] An automatic meeting minutes generation system based on OCR technology, the system includes:

[0039] Multimodal acquisition terminal: includes a 4K resolution multi-view camera, array microphones, and a hardware clock synchronization module;

[0040] Edge computing processing unit: integrates PaddleOCR engine, PaddleSpeech speech recognition engine and custom gesture detection engine;

[0041] Intelligent output module: includes a semantic template engine, rich text generator and cross-platform API interface (supports integration with Lark / DingTalk / Outlook).

[0042] The various modules of the system are connected through a time synchronization bus to achieve nanosecond-level time alignment of multimodal data.

[0043] The beneficial effects of this invention are:

[0044] (1) Intelligent generation of meeting minutes through multimodal data fusion technology: Hardware clock calibration and dynamic time warping algorithm are used to ensure nanosecond-level synchronization of multi-source data. Combined with CNN-LSTM network to detect speaker gestures in real time and dynamically adjust OCR and speech text weights to strengthen the priority of key information.

[0045] (2) Structured documents are automatically generated through configurable semantic templates and intelligent field mapping, which significantly improves standardization; BERT model is used to verify OCR coherence, combined with seating layout to correct speech errors, and low-latency processing is achieved based on edge computing devices to ensure reliability;

[0046] (3) Fine-tuning the OCR model for professional scenarios, supporting mixed recognition of handwritten or printed text, and optimizing gesture detection generalization ability through transfer learning; covering the entire process from data collection, processing and error correction to minutes generation and intelligent distribution, realizing closed-loop automation of meeting information from capture to management, with high precision, high efficiency and strong adaptability. Attached Figure Description

[0047] Figure 1 A flowchart illustrating the steps of an automatic meeting minutes generation method based on OCR technology, as provided in Example 1;

[0048] Figure 2 Here is a structural diagram of an automatic meeting minutes generation system based on OCR technology provided in Example 1;

[0049] Figure 3 This is a flowchart illustrating the execution steps of a method and system for automatically generating meeting minutes based on OCR technology, as provided in Example 2.

[0050] Figure 4 The flowchart of the method and system for automatically generating meeting minutes based on OCR technology is provided in Example 2. Detailed Implementation

[0051] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1

[0053] See Figure 1 A method for automatically generating meeting minutes based on OCR technology includes the following steps:

[0054] By deploying multi-view cameras and microphone arrays in the venue, visual image streams and surround audio streams containing whiteboard content, projected images and speaker gestures are captured simultaneously.

[0055] The visual image stream is sequentially subjected to PaddleOCR text detection, recognition, and layout analysis to obtain structured text data; the surround audio stream is subjected to end-to-end speech recognition and entity relation extraction to generate time-stamped speech text; the gesture action detection step: the speaker's gestures are detected in real time using a CNN-LSTM network model, and the gesture action type and time node corresponding to key content are marked;

[0056] Multimodal data processing: Based on the multimodal time synchronization algorithm, visual text, speech text and gesture tag data are aligned. An attention mechanism is used to build a dynamic weight allocation model, which automatically adjusts the fusion priority of OCR text and speech text in the corresponding time period according to the type of gesture action.

[0057] Following a predefined structured semantic template, the fused multimodal data is automatically populated into the structured fields of the topic categories, decision items, and action item lists, generating meeting minutes documents with key annotations.

[0058] In the gesture detection step, the CNN-LSTM network model is trained in the following way:

[0059] We used the publicly available Jester gesture dataset for pre-training to extract spatiotemporal features of gestures;

[0060] We utilize custom gesture data from meeting scenarios for transfer learning to optimize the parameters of the action classifier.

[0061] Custom gesture data includes: pointing, selection, erasing, and zooming gestures;

[0062] The output includes a confidence score for the gesture and the coordinates of the corresponding image region.

[0063] The multimodal time synchronization algorithm is as follows:

[0064] The hardware clock calibration module for the camera and microphone controls the timestamp error between the visual and audio frames within the set range.

[0065] Based on the dynamic time warping algorithm, non-linear alignment is performed on the OCR text generation time, speech recognition time, and gesture detection time.

[0066] Methods for constructing dynamic weight allocation models include:

[0067] When a "circle" or "point" gesture is detected, the weight coefficient of the corresponding OCR text area is increased to the set multiple of the voice text.

[0068] When a "wipe" gesture is detected, the filtering mechanism for the corresponding time period is triggered;

[0069] The weighting coefficients are dynamically adjusted using a reinforcement learning model trained on historical conference data.

[0070] Structured semantic templates support user-defined configurations, including:

[0071] Configurable field mapping rules: Automatically map entity types in multimodal data to template fields;

[0072] Adjustable output formatting rules: Supports generating rich text documents with table of contents, highlighted text, and associated attachments.

[0073] The method also includes a real-time error correction mechanism during execution.

[0074] Contextual semantic verification is performed on the OCR recognition results, and text coherence is detected using the BERT model, assuming a sentence... Take the CLS token vector output by BERT. As a sentence vector, the formula for calculating cosine similarity is: For recognition results with a confidence level lower than the set value, a secondary recognition is triggered. The calculation formula is as follows: ,in To adjust the hyperparameters of the slope;

[0075] in, For the OCR-recognized sentence to be verified, For contextual reference sentences, For sentences The semantic vector (the CLS token sentence vector output by the BERT model). For sentences semantic vectors (and) (Sentence vector of the same dimension as the context reference sentence).

[0076] Speaker separation processing is performed on the speech recognition results, and cross-speaker transcription errors are corrected by combining the conference room seating layout information.

[0077] The multimodal data processing is deployed on edge computing devices, specifically as follows:

[0078] NVIDIA Jetson AGX Orin is used as the edge node to achieve local real-time inference and end-to-end latency processing;

[0079] The raw data is encrypted and transmitted to the cloud via the network for model updates and historical data archiving.

[0080] The PaddleOCR processing module includes a trainable domain-adaptive model:

[0081] For specific professional fields, the text detection and recognition branches of PP-OCRv4 were fine-tuned using a small amount of labeled data;

[0082] Perform end-to-end recognition of mixed handwritten and printed text scenarios.

[0083] After the meeting minutes document is generated, an intelligent distribution step is also included:

[0084] By using NLP technology to extract key participant information from documents, to automatically generate to-do reminders and push them to the enterprise collaboration platform;

[0085] It supports generating audio summary files with timestamp indexes and linking them to the corresponding paragraphs in the meeting minutes.

[0086] See Figure 2 This paper provides an automatic meeting minutes generation system based on OCR technology, which includes:

[0087] Multimodal acquisition terminal: includes a 4K resolution multi-view camera, array microphones, and a hardware clock synchronization module;

[0088] Edge computing processing unit: integrates PaddleOCR engine, PaddleSpeech speech recognition engine and custom gesture detection engine;

[0089] Intelligent output module: includes a semantic template engine, rich text generator and cross-platform API interface (supports integration with Lark / DingTalk / Outlook).

[0090] The various modules of the system are connected through a time synchronization bus to achieve nanosecond-level time alignment of multimodal data.

[0091] Example 2

[0092] Inside the conference room, three high-definition cameras each perform their specific functions: one is aimed at the projection screen on the main stage, while the other two are aimed at the whiteboards on the left and right sides, respectively. An eight-channel circular array microphone is evenly embedded along the edge of the conference table. An edge computing device is placed in a corner of the conference room and connected to other devices via network cable. The system server is deployed in the company's server room to ensure stable operation. This scenario lays the foundation for comprehensive and accurate collection of conference data, providing environmental support for the efficient automatic generation of meeting minutes.

[0093] See Figures 3-4 The specific implementation steps are as follows:

[0094] The multimodal data acquisition process includes:

[0095] S1. Hardware Deployment and Parameter Configuration:

[0096] S1.1 Camera Deployment: When installing cameras to capture projection screens, staff repeatedly review the live feed through the management interface, fine-tuning the angle and focus until the projected content completely fills the screen without distortion, and the text is clearly legible. Simultaneously, the high-definition resolution and frame rate settings ensure that even with rapid screen transitions, there are no blurry images or ghosting issues, thus fully capturing every page of text, charts, and other information projected. For cameras capturing whiteboards, installation not only ensures the whiteboard is fully visible but also precisely adjusts the camera's tilt angle and distance based on the whiteboard's size and height. This ensures that any writing on the whiteboard is clearly captured, preventing unreadable text due to viewing angle issues and significantly improving the comprehensiveness and accuracy of visual information acquisition during meetings.

[0097] S1.2 Microphone Deployment: After installing the circular array microphones on the edge of the conference table, on-site audio recording tests will be conducted. Staff members will speak from different locations in the conference room at different volumes. The volume and sound quality of each channel will be observed through the audio processor's monitoring interface, allowing for fine-tuning of automatic gain control and noise suppression parameters. For example, if the sound is weak in a certain corner, the gain of that channel will be appropriately increased; if ambient noise filtering is incomplete, the noise suppression threshold will be further optimized. This results in clear and stable audio, effectively reducing background noise interference with speech recognition, significantly improving the accuracy of speech content acquisition, and providing high-quality audio data for accurate transcription of conference speeches.

[0098] S1.3 Clock Synchronization: After initiating the clock calibration process, the system continuously monitors the time deviation between the cameras and microphones and the standard time source. If a slight time deviation is detected on any device, a synchronization signal is automatically sent again for calibration, ensuring that the time error of all devices is controlled within a minimal range before the meeting begins. This multi-calibration and real-time monitoring mechanism guarantees a high degree of consistency in time between visual and audio data, providing a solid guarantee for the accurate fusion of subsequent multimodal data, avoiding information corruption due to time asynchrony, and greatly improving the accuracy and completeness of meeting information collection.

[0099] S2. Data Acquisition Process:

[0100] Before the S2.1 meeting begins, operators perform a comprehensive and detailed check on the system management interface. In addition to checking camera feeds and microphone recording status, they also check network connection stability and device storage capacity. If camera feeds are choppy, they immediately investigate network or device issues, such as restarting the camera or checking the network cable connection. If microphone noise is present, they readjust audio processor settings or check for microphone interference. These preventative checks and actions eliminate potential problems during data acquisition, effectively ensuring the reliability and comprehensiveness of meeting data collection.

[0101] S2.2 After clicking the "Start Recording" button, the data acquisition device will respond quickly. Simultaneously, the camera and microphone will automatically generate their respective data acquisition logs, recording information such as start time and device status. During data transmission, a redundant transmission mechanism is employed, meaning the same data is transmitted multiple times through different network paths. If one transmission path fails, the system automatically switches to a backup path, ensuring no data loss or interruption. This reliable data transmission method provides dual protection for the real-time and complete acquisition of meeting data, ensuring that every image and sound during the meeting is transmitted to the edge computing device in a timely and accurate manner, providing a solid data foundation for the real-time generation of meeting minutes.

[0102] After receiving data, the S2.3 edge computing device performs preliminary verification on temporary files. For example, it checks if the frame rate of video files is continuous and if the duration and timestamp of audio files match. If a file is found to be corrupted or missing data, a retransmission request is immediately sent to the acquisition device to ensure the integrity and usability of the stored temporary files. Furthermore, temporary files are categorized and named according to file type and acquisition time for easy retrieval and processing later. This rigorous data verification and management method further improves the accuracy and completeness of data acquisition, providing high-quality data resources for subsequent data processing.

[0103] The multimodal data processing process includes:

[0104] S3.OCR Text Recognition and Processing:

[0105] S3.1 Text Detection: When the edge computing device reads the camera image stream frame by frame, in order to improve the detection efficiency, a block detection strategy is adopted. That is, each frame of image is divided into multiple small blocks, and the text regions of each small block are detected in parallel. During the detection process, for images with complex backgrounds, such as text regions with pattern backgrounds in projection screens, the OCR recognition model will use special image preprocessing algorithms to enhance the contrast between the text and the background, making the text regions easier to be detected. At the same time, when screening invalid text boxes, in addition to considering the area and aspect ratio, the texture features within the text boxes will also be analyzed. If it is judged as a non-text pattern region, it will be excluded. Through these optimization measures, the accuracy and efficiency of text region detection are greatly improved, ensuring that all the text content in the conference visual information is accurately recognized without omission, and significantly enhancing the comprehensiveness of conference text information collection.

[0106] S3.2 Text Recognition: When preprocessing the text region image, in addition to the conventional size adjustment and noise reduction, adaptive processing will also be carried out for different fonts, font sizes and writing styles. For example, for handwritten fonts, a special stroke enhancement algorithm will be used to highlight the stroke features of the text, facilitating the recognition network to better extract the text features; for small-font text, image magnification and super-resolution processing will be carried out to improve the text clarity. During the recognition process, when encountering characters with low confidence, the system will not only mark them for verification, but also combine the context information where the characters are located to obtain parameters , and then perform semantic speculation and candidate character matching. For example, if "com[low-confidence character]er" is recognized, according to the context, it can be speculated that the character may be "put", and then the character "put" is recognized and verified again. This multi-strategy text recognition and verification mechanism effectively improves the accuracy of text recognition. Even in complex conference scenarios, various text contents can be accurately recognized, further enhancing the accuracy of conference information collection.

[0107] S3.3 Page Layout Analysis: When the page layout analysis tool is working, it will comprehensively use a variety of analysis methods. In addition to classification based on rules and machine learning, visual features such as the layout and spacing of text will also be considered. For example, for list items, features such as the distance between the bullet points and the text, and the indentation between list items will be analyzed to accurately judge; for table content, the table structure will be recognized by detecting the horizontal and vertical lines and the text distribution rules. When annotating the hierarchical relationship of text and paragraph attribution, the conference agenda information and semantic understanding results will be combined. For example, if the beginning of a certain paragraph mentions "Next, discuss the project progress" and corresponds to the "Project Progress Discussion" session in the agenda, it will be attributed to this agenda item. Through this detailed page layout analysis and content classification, the generated structured text data is well-organized, providing a good foundation for generating high-quality meeting minutes later, and greatly enhancing the structured degree and readability of the meeting minutes.

[0108] S4. Speech Recognition and Semantic Parsing:

[0109] S4.1 Audio Transcription: When denoising the audio captured by the microphone, a combination of multiple denoising algorithms is employed. First, a statistical model-based denoising algorithm is used to remove common, stable background noise. Then, for sudden, transient noises, such as door opening / closing sounds or cell phone ringtones, a time-domain analysis-based denoising method is used. In the speech recognition system, to improve adaptability to different speaking speeds and accents, a large amount of speech data with varying styles is used to train and optimize the model. During speech-to-text generation, real-time grammar checks and corrections are performed, such as automatically adding punctuation and correcting word order errors. These measures make audio transcription more accurate and fluent, enabling rapid and accurate conversion of speech to text even in complex conference speaking environments. This provides efficient and reliable audio content support for the real-time generation of meeting minutes, significantly improving the efficiency of meeting information processing.

[0110] S4.2 Entity Relationship Extraction: When extracting key entities and analyzing entity relationships, the semantic analysis model incorporates a large amount of meeting-related knowledge graphs. These knowledge graphs contain various meeting-related concepts, relationships, and attributes, such as the relationship between "project" and "responsible person" or "time node." When processing audio-text, the model matches and infers information from the text against the knowledge graph. For example, when recognizing "Zhang San is responsible for developing new features," the knowledge graph identifies "Zhang San" as the "responsible person" and "developing new features" as a "task," automatically establishing the relationship between them. Simultaneously, the model continuously learns from new meeting data and updates the knowledge graph to adapt to the needs of different meeting types. This knowledge graph-based entity relationship extraction method enables a deep understanding of meeting audio content, accurately extracting key information, making meeting minutes more concise and accurate, highlighting key points, and greatly improving the quality and practicality of meeting minutes.

[0111] S5. Gesture and motion detection:

[0112] S5.1 Model Training and Deployment:

[0113] S5.1.1 Pre-training: When pre-training using public gesture datasets, data augmentation techniques are employed to help the model better learn the spatiotemporal features of gestures. For example, random rotation, scaling, and noise addition are performed on video data to broaden the diversity of the training data. During training, the model's training status is monitored in real time, including changes in the loss function value and accuracy improvements. If overfitting is detected—that is, the model performs well on the training set but degrades on the validation set—training parameters are adjusted promptly, such as decreasing the learning rate and increasing the regularization term. Through these optimization measures, the pre-trained model can fully learn the common features of various gestures, laying a solid foundation for accurate recognition in subsequent meeting scenarios and effectively improving the generalization ability and accuracy of the gesture detection model.

[0114] S5.1.2 Fine-tuning: When collecting video data of gestures from internal company meetings, various gestures from different speakers and in different meeting scenarios are covered. When labeling the data, not only are gesture types labeled, but also detailed information such as the start and end times of the gestures and the amplitude of the movements are recorded. During fine-tuning, to accelerate training and improve training effectiveness, a transfer learning optimization strategy is adopted. This involves freezing the parameters of some network layers in the pre-trained model related to general gesture feature extraction, and only training the network layers related to the recognition of gestures specific to the meeting scenario. Simultaneously, the model's hyperparameters, such as batch size and the number of training epochs, are adjusted according to the characteristics of the training data. After meticulous fine-tuning, the model can accurately recognize common gestures in meetings, providing a reliable guarantee for accurately marking key meeting content and further improving the comprehensiveness and accuracy of meeting information collection.

[0115] S5.2 Real-time Detection Process: After the camera image is input into the gesture recognition module, a cascaded detection architecture is adopted to improve detection speed. First, a lightweight network model is used to quickly filter the image to determine if possible gesture regions exist. If a suspicious region is detected, a more complex and accurate model is used for detailed detection and recognition. When extracting the spatiotemporal features of gestures, time series analysis methods are combined. This not only analyzes the gesture image features of the current frame but also considers the gesture changes in previous and subsequent frames to more accurately determine the type of gesture. For example, for the "circle" gesture, the trajectory and shape changes of the gesture from its starting position to its ending position are observed. This real-time, efficient, and accurate gesture detection process can promptly capture every key gesture of the speaker and seamlessly integrate with other modal data, enabling meeting minutes to fully record the non-verbal emphasis points of the meeting, significantly enhancing the comprehensiveness and effectiveness of meeting information collection.

[0116] The multimodal data fusion process includes:

[0117] S6. Time Synchronization: To address time discrepancies caused by factors such as network transmission delays during time synchronization, a dynamic time compensation algorithm is employed. This algorithm predicts data transmission delays based on historical data and current network conditions, and adjusts timestamps accordingly in advance. Simultaneously, during data fusion, the synchronized data undergoes continuous time consistency verification. For example, at regular intervals, the logical consistency of content from different modalities at the same time point is checked; if inconsistencies are found, time alignment and adjustment are performed again. This dynamic adjustment and continuous verification mechanism ensures high consistency of multimodal data across the time dimension, enabling the fused data to accurately reflect the actual meeting situation. This provides strong support for generating precise meeting minutes and effectively improves the accuracy and reliability of meeting information processing.

[0118] S7. Dynamic Weight Allocation:

[0119] When a "circle" gesture is detected in S7.1, the dynamic weight allocation mechanism further analyzes the scope and duration of the circle. If the circle is large and the duration is long, it indicates that the content is likely very important, and the weight of the corresponding OCR text area will be increased accordingly. Simultaneously, to make the weight allocation more reasonable, the frequency and importance of the text in the audio text are also considered. For example, if the text in the circled area is repeatedly emphasized in the audio, the weight will be increased again. This refined weight allocation method can better highlight key information in the meeting, making the meeting minutes more focused on important content, greatly improving the readability and usability of the meeting minutes, and effectively enhancing the accuracy and effectiveness of meeting information processing.

[0120] After S7.2 detects an "erase" gesture, in addition to marking and removing data for the corresponding time period, it also performs correlation analysis on related data before and after the gesture. If the erased content is supplemented or corrected in subsequent audio or visual information, the relevant information will be retained and integrated. For example, if a piece of text on the whiteboard is erased, and the speaker mentions in audio that "the part that was just erased should be modified like this," the system will correlate the subsequent modifications with the previously erased content to ensure the completeness and accuracy of the meeting minutes. Furthermore, the generated filter log records detailed information about the erase gesture, such as time, location, and a summary of the erased content, facilitating subsequent review and auditing, further improving the reliability and traceability of meeting information processing.

[0121] When analyzing historical meeting data to adjust the parameters of the dynamic weight allocation model, the S7.3 system employs multiple data analysis methods. Besides statistically analyzing the frequency of association between different gestures and important information, it also uses cluster analysis to categorize similar meeting scenarios and gesture usage patterns, developing personalized weight allocation strategies for different types of meetings. For example, for technical solution discussion meetings and project progress reporting meetings, different weight adjustment rules are set based on their respective characteristics and common gesture usage habits. Simultaneously, meeting organizers and frequent users of meeting minutes are invited to participate in the evaluation, and parameters are further optimized based on their feedback. Through this multi-dimensional, multi-stakeholder optimization approach, the dynamic weight allocation model can continuously adapt to various meeting scenarios, improving the quality and efficiency of meeting information processing, and ensuring that meeting minutes better meet user needs.

[0122] The structured output process includes:

[0123] S8. Semantic Template Configuration:

[0124] The S8.1 system's pre-set meeting minutes templates, commonly used in technology companies, are designed with the characteristics and needs of technology industry meetings in mind. Each section of the template is meticulously planned. For example, the "Basic Meeting Information" section includes essential fields such as meeting name, time, location, chairperson, and participants, facilitating a quick overview of the meeting. The "Agenda Discussion" section follows the meeting agenda, with sub-fields for topics, discussion content, and relevant participants' remarks, enabling detailed recording of the meeting discussion process. This pre-set template provides a scientific and reasonable basic structural framework for meeting minutes, ensuring a uniform format and standardization, greatly improving the standardization and readability of meeting minutes, and facilitating quick reading and management by users.

[0125] When users modify templates in the S8.2 interface, the system provides an intuitive and convenient operation method. For example, fields can be added or deleted with simple drag-and-drop operations, and double-clicking a field allows modification of its name and format settings. When adding custom fields, the system automatically prompts relevant setting options, such as field type (text, date, and number, etc.), whether it is required, and display order. After modification, the system will preview the template effect in real time, allowing users to make adjustments. This flexible and easy-to-use template customization function meets the personalized needs of different users and different meeting types for minutes format. Whether it is a small internal departmental meeting or a large cross-departmental project meeting, meeting minutes that meet the requirements can be generated through custom templates, significantly enhancing the system's adaptability and flexibility.

[0126] When setting field mapping rules in S8.3, the system employs a combination of intelligent matching and manual adjustment. First, the system automatically performs preliminary field mapping based on the type and semantic features of entities in the multimodal data. For example, identified time-related entities are automatically mapped to time-related fields in the template. For complex or ambiguous cases, users can manually adjust the mapping relationships. Simultaneously, the system supports batch setting of mapping rules, allowing multiple fields of the same type to be set at once. Furthermore, the system saves the user-set mapping rules, automatically applying them the next time similar meeting data is encountered, improving the efficiency and accuracy of meeting minutes generation and making the meeting minutes more structured and organized.

[0127] S9. The document acquisition process includes:

[0128] When the S9.1 system populates the fused multimodal data according to the template format, it performs rigorous data validation and format conversion. For data obtained from different modalities, its completeness and accuracy are checked, such as verifying the completeness of OCR-recognized text and the absence of missing paragraphs in speech-recognized text. During the population process, the data is converted accordingly based on the format requirements of the template fields. For example, time-related data is converted to a specified date and time format, and list item data is formatted according to the template's list format. Simultaneously, to ensure the accuracy of the data population, a second check is performed on the populated content to ensure that each field is accurately populated with the corresponding data. This rigorous data processing and population method results in complete, accurate, and well-organized meeting minutes, improving the quality and readability of the minutes and facilitating a comprehensive understanding of the meeting proceedings for users.

[0129] When S9.2 automatically highlights key content, the system not only relies on gesture markings and voice emphasis but also incorporates semantic analysis results. For example, key decisions and important goals mentioned in the meeting will be highlighted in red based on semantic importance, even without obvious gestures or voice emphasis. When automatically linking attachments, the system identifies the relevance of attachments to the meeting content. For instance, if a slide in a PowerPoint presentation is related to a topic discussed in the meeting, it will be accurately linked to that slide in the corresponding location in the meeting minutes. When adding timestamp-indexed audio summaries, the system automatically generates appropriate index nodes based on the chapter divisions and distribution of key content in the meeting. For example, indexes are set at the beginning and end of each topic discussion, allowing users to quickly locate audio segments of interest. The combined use of these functions further highlights the key points of the meeting, enriches the form and content of the meeting minutes, and greatly improves the ease of use of the meeting minutes.

[0130] The method described in this embodiment achieves intelligent generation of meeting minutes through multimodal data fusion technology (integrating visual, speech, and gesture signals): It utilizes hardware clock calibration and dynamic time warping algorithms to ensure nanosecond-level synchronization of multi-source data; combines a CNN-LSTM network to detect speaker gestures in real time (such as selection and erasing) and dynamically adjusts the weights of OCR and speech text to enhance the priority of key information; it automatically generates structured documents (including table of contents, highlighted key points, and attachment associations) through configurable semantic templates and intelligent field mapping, significantly improving standardization; it uses a BERT model to verify OCR coherence, corrects speech errors based on seating layout, and achieves low-latency processing based on edge computing devices to ensure reliability; it fine-tunes the OCR model for professional scenarios, supports mixed recognition of handwritten and printed text, and optimizes gesture detection generalization capabilities through transfer learning; the entire process covers data acquisition, processing, and error correction to minute generation and intelligent distribution (such as to-do push and audio summaries), achieving closed-loop automation of meeting information from capture to management, combining high precision, high efficiency, and strong adaptability.

[0131] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for automatically generating meeting minutes based on OCR technology, characterized in that, Includes the following steps: By deploying multi-angle cameras and microphone arrays in the venue, visual image streams and surround audio streams containing whiteboard content, projected images, and speaker gestures are captured simultaneously. The visual image stream is sequentially subjected to PaddleOCR text detection, recognition, and layout analysis to obtain structured text data; the surround audio stream is subjected to end-to-end speech recognition and entity relation extraction to generate time-stamped speech text; the gesture action detection step: the speaker's gestures are detected in real time using a CNN-LSTM network model, and the gesture action type and time node corresponding to key content are marked; Multimodal data processing: Visual text, speech text, and gesture-tagged data are aligned based on a multimodal time synchronization algorithm. An attention mechanism is used to construct a dynamic weight allocation model, denoted as [model name missing]. Attention weights The text embedding vector for OCR recognition. Standard speech-to-text embedding vector; For the embedding dimension, the corresponding formula is: ,in It is a weight matrix. It is a bias term. It uses the sigmoid function, and then automatically adjusts the fusion priority of OCR text and speech text for the corresponding time period based on the type of gesture. It is the fused text representation, calculated using the following formula: ; According to the predefined structured semantic template, the fused multimodal data is automatically populated into the structured fields of the topic classification, decision items and action items list to generate meeting minutes documents with key annotations; The method for constructing the dynamic weight allocation model includes: When a "circle" or "point" gesture is detected, the weight coefficient of the corresponding OCR text area is increased to the set multiple of the voice text. When a "wipe" gesture is detected, the filtering mechanism for the corresponding time period is triggered; The weighting coefficients are dynamically adjusted using a reinforcement learning model trained on historical conference data. It also has a real-time error correction mechanism: Contextual semantic verification is performed on the OCR recognition results, and text coherence is detected using the BERT model, assuming a sentence... Take the CLS token vector output by BERT. As a sentence vector, the formula for calculating cosine similarity is: For recognition results with a confidence level lower than the set value, a secondary recognition is triggered. The calculation formula is as follows: ,in To adjust the hyperparameters of the slope; in, For the OCR-recognized sentence to be verified, For contextual reference sentences, For sentences semantic vectors, For sentences semantic vector; Speaker separation processing is performed on the speech recognition results, and cross-speaker transcription errors are corrected by combining the conference room seating layout information.

2. The method for automatically generating meeting minutes based on OCR technology according to claim 1, characterized in that, In the gesture detection step, the CNN-LSTM network model is trained in the following manner: We used the publicly available Jester gesture dataset for pre-training to extract spatiotemporal features of gestures; By leveraging custom gesture data from meeting scenarios for transfer learning, the parameters of the action classifier are optimized and updated. ,in Is In the next iteration, the first-order matrix of the gradient is estimated. It is a second-order estimate of the gradient. For learning rate, It is a small constant; Custom gesture data includes: pointing, selection, erasing, and zooming gestures; The output includes a confidence score for the gesture and the coordinates of the corresponding image region.

3. The method for automatically generating meeting minutes based on OCR technology according to claim 1, characterized in that, The multimodal time synchronization algorithm is specifically as follows: The hardware clock calibration module for the camera and microphone controls the timestamp error between the visual and audio frames within the set range. Based on the dynamic time warping algorithm, non-linear alignment is performed on the OCR text generation time, speech recognition time, and gesture detection time.

4. The method for automatically generating meeting minutes based on OCR technology according to claim 1, characterized in that, The structured semantic template can be customized by the user, including: Configurable field mapping rules: Automatically map entity types in multimodal data to template fields; Adjustable output formatting rules: Generate rich text documents with a table of contents, highlighted text, and associated attachments.

5. The method for automatically generating meeting minutes based on OCR technology according to claim 1, characterized in that, The multimodal data processing is deployed on an edge computing device, specifically: NVIDIA Jetson AGX Orin is used as the edge node to achieve local real-time inference and end-to-end latency processing; The raw data is encrypted and transmitted to the cloud via the network for model updates and historical data archiving.

6. For specific professional fields, the text detection and recognition branches of PP-OCRv4 are fine-tuned using labeled data; Perform end-to-end recognition of mixed handwritten and printed text scenarios.

7. The method for automatically generating meeting minutes based on OCR technology according to claim 1, characterized in that, After the meeting minutes document is generated, an intelligent distribution step is also included: By using NLP technology to extract key participant information from documents, to automatically generate to-do reminders and push them to the enterprise collaboration platform; Generate an audio summary file with timestamp index and associate it with the corresponding paragraph of the meeting minutes.

8. A meeting minutes automatic generation system based on OCR technology, employing the meeting minutes automatic generation method based on OCR technology as described in any one of claims 1-7, characterized in that, The system includes: Multimodal acquisition terminal: includes multi-view camera, array microphone and hardware clock synchronization module; Edge computing processing unit: integrates PaddleOCR engine, PaddleSpeech speech recognition engine and custom gesture detection engine; Intelligent output module: includes a semantic template engine, a rich text generator, and cross-platform API interfaces; The various modules of the system are connected through a time synchronization bus to achieve nanosecond-level time alignment of multimodal data.

Images