Multi-modal data format intelligent conversion method

Abstract

The invention discloses a multi-modal data format intelligent conversion method, and belongs to the field of data processing. The method comprises the following steps: firstly identifying a data type, then carrying out optimized OCR conversion (improving handwritten form accuracy) on an image, carrying out dual-model fusion transcription and sentiment analysis on audio, carrying out key frame extraction and multi-modal fusion processing on video, and finally unifying a data format and adding an associated ID (Identity). The method solves the problems of single function, low complex format accuracy, data fragmentation and no follow-up adaptability in the prior art, can process multi-modal data in a one-stop manner, improves the conversion quality and efficiency, adapts to a follow-up data capitalization module, and is suitable for government and enterprise multi-modal data conversion scenes.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a multimodal data format intelligent conversion method integrating OCR, speech-to-text, and video structuring processing, applicable to scenarios involving the conversion of various types of unstructured data to structured data in government and enterprise units. Background Technology

[0002] During the process of data assetization, government and enterprise units have accumulated a large amount of multimodal unstructured data (such as scanned copies of paper documents, audio files, and surveillance videos). This type of data needs to be converted into a structured format before it can be used for annotation and analysis, but existing technologies have the following drawbacks: 1. Limited functionality: Existing tools are mostly designed for a single modality. For example, OCR tools only support image-to-text conversion, and speech-to-text tools only support audio-to-text conversion. This requires switching between multiple tools to process multimodal data, which is cumbersome and inefficient. 2. Low accuracy for complex formats: Existing tools have an accuracy rate of less than 85% for converting complex format data (such as handwritten paper scans, low-resolution images, and audio in noisy environments). For example, the OCR recognition error rate for handwritten official document scans is high, making them unusable directly. 3. Fragmented data after conversion: The data formats of converted images, audio, and video are not uniform (e.g., text is stored in TXT format, image features are stored in JSON format), and there are no associated identifiers, making it impossible to form a structured dataset. Subsequent manual integration is required, which is costly. 4. Lack of subsequent adaptability: The converted data is not optimized according to the format requirements of the subsequent governance and annotation modules, and requires secondary processing before it can be connected to the data assetization platform, which affects the efficiency of the process.

[0003] The aforementioned defects result in low efficiency and poor quality of multimodal data conversion, hindering the process of data assetization. Summary of the Invention

[0004] 3.1 Technical problems to be solved This invention aims to solve the problems of "single function, low accuracy of complex formats, data fragmentation, and lack of subsequent adaptability" in the prior art, and provides a method for intelligent conversion of multimodal data formats that integrates multimodal conversion function, improves the accuracy of complex formats, unifies data structure, and adapts to subsequent modules.

[0005] 3.2 Technical Solution The technical solution of the present invention is as follows, and its steps include: S1: Multimodal data type recognition A multimodal data format recognition model is constructed. The model is trained based on a deep learning classification algorithm (such as ResNet50). After inputting the data to be converted, it automatically identifies the data type: text (TXT / PDF), image (JPG / PNG / BMP), audio (WAV / MP3), and video (MP4 / AVI), with a recognition accuracy of ≥99%. S2: Image data conversion (including scanned paper documents) For image data, an OCR model optimized with a "CNN + attention mechanism" is used to perform the conversion: Preprocessing: Gaussian filtering is performed on the blurred image to remove noise, and skeleton extraction and enhancement are performed on the handwritten image; Text detection: The EAST algorithm is used to detect text regions in images, with a localization accuracy of ≥98%. Text recognition: The model uses CRNN + attention mechanism to recognize text. For government and enterprise special fonts (such as Songti for official documents), it is fine-tuned through domain samples. The accuracy rate of handwritten font recognition is ≥92%, and the accuracy rate of printed font recognition is ≥98%. Output: Generate a structured JSON file from the recognition results in the format of "image ID-text content-text coordinates"; S3: Audio Data Conversion For audio data, a transformation is performed by fusing the Whisper model and the Wav2Vec2 model: Preprocessing: Noise reduction is performed on noisy audio (using spectral subtraction), and long audio (>30 minutes) is segmented. Speech-to-text: Transcription is performed using the Whisper-large model, with a transcription accuracy of ≥98%; Sentiment analysis: The Wav2Vec2 model is used to extract audio sentiment features and output "positive / negative / neutral" sentiment labels, with a sentiment recognition accuracy of ≥90%. Output: Generate a structured CSV file in the format of "Audio ID-Transcription Text-Emotion Tag-Timestamp"; S4: Video Data Conversion For video data, perform transformation using FFmpeg tools and a multimodal model: Keyframe extraction: FFmpeg is used to extract video keyframes at 1 frame / second, and blurry frames (frames with a resolution <0.8) are filtered out. Keyframe processing: Perform OCR conversion on the keyframes in step S2 to recognize text information in the image; Audio processing: Extract the audio track from the video and perform transcription and sentiment analysis in step S3; Structured description: By integrating keyframe text and audio transcription results, a structured description JSON file is generated, consisting of "video ID - keyframe image - transcribed text - sentiment tag - timeline". S5: Data Format Standardization and Adaptation The conversion results from steps S2-S4 are uniformly encoded into UTF-8 format, and a standardized data structure is generated according to the input requirements of subsequent data governance and annotation modules. Text fields: uniform length limit (single field ≤ 2000 characters); Association Identifier: Add a unified association ID to multimodal data (such as "conference video + meeting recording") from the same data source; Output formats: Supports CSV, JSON, and Parquet formats, and can be directly integrated into the governance module of the data assetization platform.

[0006] 3.3 Beneficial Effects 1. Integrated multimodal conversion: One-stop processing of image, audio, and video data, eliminating the need to switch tools and improving conversion efficiency by 40%; 2. High accuracy in complex formats: The optimized OCR model improves the accuracy of handwritten and blurred image recognition to ≥92%, and the fusion model improves the accuracy of audio transcription to ≥98%, meeting the high-precision conversion needs of government and enterprises; 3. Unified data structuring: Unified coding and data structure, addition of association identifiers, avoids data fragmentation, eliminates the need for subsequent manual integration, and reduces costs by 30%; 4. Adaptation to subsequent modules: The conversion results can be directly connected to the governance and annotation modules without secondary processing, improving the efficiency of the data assetization process by 25%.

Claims

1. A method for intelligent conversion of multi-modal data formats, characterized in that, The method comprises the following steps: S1: multi-modal data type recognition A multi-modal data format recognition model based on ResNet50 algorithm is constructed. After inputting the data to be converted, the data type is automatically recognized as text, image, audio or video, and the recognition accuracy is ≥99%; S2: image data conversion For image data, an OCR model optimized by "CNN+attention mechanism" is used for conversion: first, denoising and enhancement preprocessing, then EAST algorithm is used to detect the text area, and finally CRNN+attention mechanism model is used to recognize the text, output "image ID-text content-text coordinate" JSON file, handwriting recognition accuracy ≥92%, printed matter recognition accuracy ≥98%; S3: audio data conversion For audio data, the Whisper-large model and Wav2Vec2 model are fused to perform conversion: first, denoising and segmentation preprocessing, then using the Whisper model to transcribe text (accuracy ≥98%), and using the Wav2Vec2 model to output emotion labels, generating "audio ID-transcribed text-emotion label-timestamp" CSV file; S4: video data conversion For video data, FFmpeg is used to extract key frames at 1 frame per second and filter blurred frames. The OCR conversion of step S2 is performed on the key frames, and the processing of step S3 is performed on the video audio track, generating "video ID-key frame image-transcribed text-emotion label-time axis" JSON file; S5: data format unification and adaptation The conversion results are uniformly encoded into UTF-8 format, and a unified association ID is added. Output in CSV, JSON or Parquet format, which can be directly connected to the data governance module.

2. The method of claim 1, wherein, In step S1, the text data includes TXT, PDF format, the image data includes JPG, PNG, BMP format and paper scans, the audio data includes WAV, MP3 format, and the video data includes MP4, AVI format.

3. The method of claim 1, wherein, In step S2, the preprocessing includes Gaussian filter denoising for blurred images and skeleton extraction enhancement for handwritten images. The text area detection uses the EAST algorithm, and the positioning accuracy is ≥98%.

4. The method of claim 1, wherein, In step S3, the segment length of long audio segmentation processing is 10-30 minutes, and the emotion labels include positive, negative and neutral, with an emotion recognition accuracy of ≥90%.

5. The method of claim 1, wherein, In step S4, the blurred frame filtering standard is clarity <0.8, and the clarity is calculated by the Laplacian variance algorithm.

6. The method of claim 1, wherein, In step S5, the unified association ID is in UUID format, and the text field length limit is ≤2000 characters per field.