Heterogeneous data conversion method and system based on multi-modal large model

Abstract

The application discloses a kind of based on multimodal big model's heterogeneous data conversion method and system, to solve the problems existing in traditional heterogeneous data processing, such as single mode limitation, high migration cost etc. Method is first to image, text, audio and video etc. Multi-source heterogeneous data are carried out format adaptation and type pre-processing. Then through ViT, BERT and Transformer cross attention mechanism, realize the deep analysis of visual, text, audio features and cross-modal semantic association, establish "visual-text-audio" semantic mapping. Again based on dynamic prompt engineering, guide multimodal big model to generate structured data according to preset Schema zero sample. Then through Drools rule engine error correction and format normalization, ensure that data meet business standards. Finally support JSON / XML / database table etc. Multi-format output, while associated with original data storage, realize data traceability. The application breaks through the limitation of single mode processing, realizes cross-modal data deep association.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and data engineering, and particularly relates to a heterogeneous data conversion method and system based on a multi-modal large model. BACKGROUND

[0002] Under the current digital wave, the data assets of enterprises and institutions exhibit multi-source and heterogeneous characteristics, covering various forms such as text reports, table images, industrial sensor videos, medical images, and financial contract scans. Most business data exists in unstructured or semi-structured forms, while traditional data processing systems rely more on structured data such as relational database tables to support business decisions, resulting in difficulties in intercommunication and integration of data of different modalities and formats, and failing to fully release the value of data.

[0003] With the deep application of artificial intelligence technology in fields such as financial risk control, medical diagnosis, and industrial quality inspection, the demand for real-time conversion and unified analysis of heterogeneous data has surged. For example, financial institutions need to simultaneously process visual modal customer ID images, text modal credit investigation text reports, and audio modal transaction flow audio records to complete credit approval. In industrial scenarios, device vibration audio, fault snapshot images, and production log texts need to be associated to achieve predictive maintenance. However, existing technical systems are difficult to efficiently meet the data conversion needs of such cross-modal and cross-format data, and the main problems are:

[0004] (1) Limitations of single-modal processing technology. Traditional NLP tools such as BERT and ROBERTA can only parse pure text semantics. Conventional OCR (Optical Character Recognition) tools can only extract text content from images and lack semantic understanding capabilities. Traditional ASR (Automatic Speech Recognition) can only convert audio to text scripts and cannot simultaneously capture emotional features such as tone fluctuations when customers complain, nor can it perform linked analysis with scene images corresponding to the audio, such as screenshots during customer service calls.

[0005] (2) High dependence on rule templates. Existing tools such as Apache Camel and Talend require customized conversion rules for specific data formats, such as regular expression matching of text fields and XML Schema definition of data structures. When data formats change, such as adjustment of bank statement table column order or update of medical report templates, rules need to be rewritten, which has poor generalization ability and high maintenance cost. For example, if a retail enterprise adds 10 different versions of product labels, it needs to develop conversion templates for each version, making it difficult to adapt to rapid business iteration.

[0006] (3) Lack of cross-modal association capability: Traditional solutions mostly use the mode of "single-modal processing + splicing integration", which cannot establish semantic association between different modal data. For example, in the government certificate verification scene, the existing system can extract the text including the name and ID number in the ID image and the similarity score of the face recognition result, but cannot verify whether the "ID photo and face image are of the same person", which is the core logic. Sometimes, it needs manual secondary verification, which is low in efficiency.

[0007] (4) High dependence on labeled data and high migration cost: Special small models such as NER entity recognition models and image classification models for specific industries need a large amount of labeled data for training, and the model generalization is limited. The contract information extraction model trained in the financial field cannot be directly migrated to the medical record conversion task in the medical field, and needs to be re-labeled with medical data and trained, which significantly increases the time and labor costs.

[0008] Therefore, there is a need for a heterogeneous data conversion method and system with cross-modal association capability, zero-shot adaptability and business post-processing mechanism to solve the problems in the prior art. SUMMARY

[0009] In view of the above defects of the prior art, the present application breaks through the limitations of traditional heterogeneous data conversion technology by innovative technical architecture and process design, adopts deep association of cross-modal data, and based on dynamic prompt engineering and multi-modal large model, significantly improves the data processing efficiency, accuracy, adaptability and business value release.

[0010] The first object of the present application proposes a heterogeneous data conversion method based on a multi-modal large model, comprising the following steps:

[0011] S1. Data input and preprocessing: For multi-source heterogeneous input data, perform format normalization, and perform preliminary information extraction according to data type differentiation; the multi-source heterogeneous input data includes image type, text type, audio and video type;

[0012] S2. Multi-modal joint analysis: Through visual feature extraction, text semantic analysis and cross-modal association process, semantic mapping between different modal data is established;

[0013] S3. Zero-shot structured generation: based on dynamic prompt engineering, guide the multi-modal large model to output structured data according to the preset Schema;

[0014] S4. Self-adaptive post-processing: with the help of error correction function and normalization function, solve the problem of abnormal value and inconsistent format in the large model generation result, so that the structured data meets the business standard;

[0015] S5. Output and storage according to user requirements: structured data meeting business standards are output and stored according to user requirements, while retaining the original heterogeneous data link, forming a traceable closed loop from structured data to original data.

[0016] Preferably, the step S1 comprises:

[0017] S11. Multi-source heterogeneous input data format access, data type classification and adaptation; the image class includes JPEG, PNG, TIFF format; the text class includes PDF, DOCX, TXT format; the audio and video class includes WAV, MP3, MP4, AVI format;

[0018] S12. Data preprocessing operation according to type: format unification, quality optimization and preliminary text positioning are carried out on the image class, the format unification can convert non-standard format into JPEG format, the quality optimization uses Gaussian filter denoising and adaptive threshold binarization to process fuzzy images, and the preliminary text positioning detects text area by Tesseract OCR tool and outputs related information; format analysis, noise filtering and entity preliminary marking are carried out on the text class, the format analysis extracts text content and format information by Apache PDFBox and Python-docx tools, the noise filtering eliminates redundant characters and corrects OCR recognition errors, and the entity preliminary marking marks key entities by jieba segmentation combined with self-defined industry dictionary; audio transcription and video key frame extraction are completed on the audio and video class, and "frame-text-time" mapping construction is completed, audio processing transcribes text by Whisper model and retains timestamp and extracts audio features, video processing extracts key frames according to frequency and performs image preprocessing, and binds audio transcription text with corresponding frame timestamp.

[0019] Preferably, the step S2 comprises:

[0020] S21. Visual feature extraction: Mask R-CNN is used to identify structured elements in the image and output bounding box coordinates and class labels, and then the image is divided into 16x16 pixel blocks, and the linear projection layer of ViT is used to map to 768-dimensional embedding space to generate image block sequence features, and cell position features are additionally extracted for table regions; the structured elements include tables, charts and icons;

[0021] S22. Text semantic analysis: preprocessed text is input into BERT model to generate 768-dimensional word-level embedding vectors, BiLSTM model is used to strengthen long text semantic association, and industry-specific NER model is used to identify core fields in the text;

[0022] S23. Cross-modal association: With the help of the Transformer cross-attention mechanism, the cosine similarity between the visual block embedding and the text word embedding is calculated to generate a "visual-text association matrix". In the cross-attention layer, the multi-head attention mechanism is used to strengthen the association of high similarity features. The cross-modal association result is fused with the global context.

[0023] Preferably, the step S3 comprises:

[0024] S31. Target Schema parsing and prompt template generation: Read the user's preset structured output format, extract field name, data type, constraint condition, and generate a special prompt template containing "role definition - task description - format constraint - rule description" according to the application scenario; the structured output format includes JSON Schema, database table structure;

[0025] S32. Prompt word optimization and multi-modal feature splicing: By testing and comparing the generation accuracy under different prompt structures, the prompt word weight is adjusted combined with reinforcement learning algorithm, and the preprocessed multi-modal features and optimized prompt words are spliced to form a large model input sequence;

[0026] S33. Large model structured generation and candidate screening: The multi-modal large model fills in the values field by field to generate initial structured data according to the input sequence, uses beam search to generate candidate structured results, calculates the confidence of each candidate result, and selects the candidate result with the highest confidence as the initial output; the multi-modal large model is GPT-4V or LLaVA.

[0027] Preferably, the step S4 comprises:

[0028] S41. Based on the Drools rule engine, configure industry-specific verification rules to automatically detect abnormal fields, set a confidence threshold, and mark fields with a confidence lower than the threshold as "to be manually reviewed";

[0029] S42. Perform format standardization, unify different formats of time into ISO 8601 format, and unify different representations of numerical values into floating-point numbers with fixed decimal places; perform unit unification;

[0030] S43. Provide a manual review interface, display "original multi-modal data - abnormal field - candidate value" through a Web-based visualization review interface, and allow staff to manually correct.

[0031] Preferably, the step S5 comprises:

[0032] S51. Structured data output; the structured data includes structured file formats and database formats; the structured file formats include JSON format, XML format, CSV format; for database format output, SQL statements are generated, structured data is written to a relational database, and indexes are established for related fields;

[0033] S52. Data storage optimization: a table splitting strategy is adopted when writing structured data to a relational database; the original heterogeneous data before preprocessing is stored to an object storage service, and the storage path and metadata of the original data are retained in the structured data; log information of the entire conversion process is recorded in detail;

[0034] S53. Output verification: the system checks the integrity and consistency of the output data, and if inconsistent data is found, a retry mechanism is triggered to re-execute the processes of steps S3 to S5.

[0035] The second object of the application proposes a heterogeneous data conversion system based on a multi-modal large model, which includes an input adaptation layer, a multi-modal analysis layer, a structured generation layer, an adaptive post-processing layer, and an output storage layer:

[0036] (1) Input adaptation layer: including format adaptation unit and type-based preprocessing unit; the format adaptation unit supports multi-source heterogeneous data mainstream format access and converts non-standard formats; the type-based preprocessing unit performs differential preprocessing operations on different types of data, optimizes data quality, and extracts preliminary information;

[0037] (2) Multi-modal analysis layer: composed of visual feature extraction unit, text semantic analysis unit and cross-modal correlation unit, the visual feature extraction unit analyzes image structured elements and visual features, the text semantic analysis unit understands text semantics and identifies key entities, and the cross-modal correlation unit establishes cross-modal semantic correlation;

[0038] (3) Structured generation layer: including Schema analysis unit, prompt engineering unit and large model generation unit, the Schema analysis unit extracts preset output Schema information, the prompt engineering unit generates and optimizes prompt templates, and the large model generation unit calls multi-modal large models to generate and filter structured data;

[0039] (4) Adaptive post-processing layer: including error correction unit, normalization unit and manual review unit, the error correction unit checks and marks abnormal and low confidence data, the normalization unit unifies data format and unit, and the manual review unit provides an interface to support manual data correction;

[0040] (5) Output storage layer: composed of format output unit, storage optimization unit and output verification unit, the format output unit outputs multi-format data, the storage optimization unit stores data in layers and records logs, and the output verification unit checks data consistency and triggers a retry mechanism.

[0041] Preferably, in the sub-type preprocessing unit, image preprocessing adopts Gaussian filter denoising, adaptive threshold binary optimization image, and locates the text area through Tesseract OCR tool; text preprocessing extracts text information by ApachePDFBox and Python-docx tools, and adopts jieba segmentation combined with industry dictionary to mark key entities; audio and video preprocessing transcribes audio and extracts intonation features by using Whisper model, and extracts key frames of video according to frequency.

[0042] Preferably, the cross-modal association unit generates an association matrix by calculating the cosine similarity of visual block embedding and text word embedding, and strengthens the association of high similarity features in the cross attention layer by using the multi-head attention mechanism, and integrates the association results combined with the global context.

[0043] Preferably, the storage optimization unit stores structured data in tables according to scenarios, and the format output unit establishes an index for key fields of the database.

[0044] The present application has the following beneficial effects:

[0045] (1) Breakthrough single-modal processing limitations, realize cross-modal data deep association. The present application realizes cross-modal data deep association by using the multi-modal joint analysis architecture, relying on ViT visual feature extraction, BERT text semantic analysis and Transformer cross attention association technology, constructing "visual-text-audio" semantic mapping, solving the problem of cross-modal data fragmentation, improving the data association integrity compared with the traditional scheme, and providing complete data support for multi-dimensional business analysis.

[0046] (2) Zero sample adaptation reduces migration cost and greatly improves business flexibility. The present application is based on dynamic prompt engineering and pre-trained multi-modal large model, generates a scenario-based prompt template through a preset Schema, and realizes new scene adaptation without fine-tuning the model. The deployment cycle of the added heterogeneous data conversion scene is greatly shortened, especially suitable for small and medium-sized enterprises or emerging fields and other scenes lacking of labeled data.

[0047] (3) Self-adaptive post-processing guarantees data quality and meets business standard requirements. The present application realizes industry-specific verification based on Drools rule engine through the cooperative action of the error correction module and the normalization module, combines format standardization and confidence filtering, and the accuracy of structured data is high.

[0048] The application constructs an input preprocessing, joint analysis, structured generation, post-processing, storage closed-loop automation process, through cooperation and optimization between modules, the business response speed is significantly improved, the business value of heterogeneous data is released, and the intelligent upgrading of digital business is promoted. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The method steps of the embodiment of the application are shown in the following. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the application.

[0051] Embodiment one

[0052] The heterogeneous data conversion method based on the multi-modal large model proposed in this embodiment can efficiently complete the conversion from unstructured or semi-structured data covering images, texts, audio and video, etc. to standardized structured data including JSON, XML, database tables, etc. by means of cross-modal fusion technology, dynamic prompt engineering and business verification mechanism. Figure 1 As shown in the figure, the specific steps are as follows:

[0053] S1, data input and preprocessing. This step is aimed at multi-source heterogeneous input data such as images, texts, audio and video, and carries out format regularization and information preliminary extraction operation. The operation will be differentiated according to the data type.

[0054] S11, input data type adaptation. The system supports the access of mainstream heterogeneous data formats, and the specific classification is as follows:

[0055] 1. Image type: including JPEG, PNG, TIFF and other formats, covering scanned documents such as contracts, test sheets, screenshots of commodity labels, certificates, and printed matter such as reports and invoices.

[0056] 2. Text type: related to editable or scanned PDF, DOCX, TXT and other formats, including pure text reports such as credit reports, and formatted text such as medical records with tables.

[0057] 3. Audio and video type: audio formats include WAV, MP3, such as customer service call recording, transaction flow broadcast, etc.; video formats include MP4, AVI, such as industrial equipment monitoring video, government window business video, etc.

[0058] S12, type-based preprocessing operation.

[0059] 1. Image preprocessing.

[0060] Format unification: Convert non-standard formats such as TIFF to JPEG format to ensure compatibility of subsequent processing steps and avoid processing interruptions or errors due to format differences.

[0061] Quality optimization: Use Gaussian filtering technology to remove noise generated during scanning, and use adaptive threshold binarization method to process fuzzy images such as handwritten medical records, significantly improving the clarity of the text and providing a better image basis for subsequent text recognition.

[0062] Initial text positioning: Use Tesseract OCR tool to detect text areas in images, output text candidate box coordinates and preliminary recognition results, such as "blood sugar: 5.6 mmol / L", to lay the foundation for subsequent implementation of visual and text association.

[0063] 2. Text preprocessing.

[0064] Format analysis: For PDF and DOCX files, use Apache PDFBox and Python-docx tools to extract text content and format information, such as table row and column structure, title hierarchy, etc., to accurately grasp the organization form of the text.

[0065] Noise filtering: Remove redundant characters in the text, such as line breaks and special symbols; for OCR recognition results of scanned PDF, correct spelling errors, such as correcting "blood sugar" misrecognized as "blood Tang" according to the dictionary, to ensure the accuracy of the text content.

[0066] Entity preliminary labeling: Use jieba segmentation combined with a lightweight NLP tool based on a custom industry dictionary to label key entities, such as "invoice number" and "amount" in the financial field, and "patient ID" and "test indicators" in the medical field, to prepare for subsequent in-depth analysis of text semantics.

[0067] 3. Audio and video preprocessing.

[0068] Audio processing: Use the Whisper model to transcribe audio into text scripts while retaining timestamp information, such as "00:01:20 - transaction amount: 15800 yuan"; in addition, extract audio features such as tone fluctuations to provide data support for subsequent emotional correlation analysis.

[0069] Video processing: Key frames of the video can be extracted at a frequency of "1 frame per second" to avoid data redundancy; image preprocessing operations are performed on the extracted key frames; and audio transcription text is bound with the timestamp of the corresponding frame to construct a "video frame - text - time" mapping table, such as forming an association record like "00:05:30 frame - device vibration anomaly" in an industrial monitoring video.

[0070] S2, multi-modal joint analysis. In this step, through a three-level process of "visual feature extraction - text semantic analysis - cross-modal association", the limitations of single-modal processing are broken down, and semantic mapping between different modal data is established. The specific implementation is as follows:

[0071] S21, visual feature extraction. A pre-trained ViT (Vision Transformer) model and Mask R-CNN tool are used to deeply analyze the structure and visual elements in the image.

[0072] 1. Structured element detection: Mask R-CNN is used to identify structured elements such as tables, charts, icons, etc. in the image, and output the bounding box coordinates of these elements, such as the row height, column width of the table, and category labels such as "medical test report table" and "line chart", to clearly define the location and attributes of various structured elements in the image.

[0073] 2. Visual feature encoding: The image is divided into 16x16 pixel blocks, and these pixel blocks are mapped to a 768-dimensional embedding space through the linear projection layer of ViT to generate image block sequence features; For the detected table area, additional cell position features such as "3rd row, 2nd column cell" are extracted to create conditions for subsequent association with text values.

[0074] S22, text semantic analysis. Based on the BERT model and BiLSTM model, deep understanding and key information extraction of text semantics are realized.

[0075] 1. Text feature encoding: The preprocessed text is input into the BERT model to generate 768-dimensional word-level embedding vectors, accurately capturing the context semantics of words in specific contexts, such as the specific meaning of "blood pressure" in the sentence "blood pressure: 120 / 80 mmHg".

[0076] 2. Context modeling: The BiLSTM model is used to process the word embedding sequence to strengthen the semantic association in long text, such as clearly defining the ownership relationship between "test date: 2024-09-01" and the subsequent "blood sugar: 5.6 mmol / L" in a medical report, avoiding information fragmentation.

[0077] 3. Key Entity Recognition: Combine industry-specific NER (Named Entity Recognition) models, such as financial invoice NER and medical indicator NER, to accurately identify core fields in the text. For example, in "Invoice Number: INV - 20240901," accurately distinguish "Invoice Number" as the field name and "INV - 20240901" as the field value.

[0078] S23, Cross-modal association. With the help of Transformer cross-attention mechanism, establish accurate mapping between visual elements and text semantics, solve the problem of visual position and text meaning disconnection.

[0079] 1. Association matrix construction: Calculate the cosine similarity between visual block embeddings such as table cell features and text word embeddings such as "blood sugar" and "5.6 mmol / L", generate "visual-text association matrix", quantify the degree of association between the two.

[0080] 2. Semantic alignment: In the cross-attention layer, use multi-head attention mechanism to strengthen the association of high similarity feature pairs, for example, the similarity between "blood pressure" text and the cell where "120 / 80 mmHg" is located in the table is 0.92, preferentially establish mapping relationship for them, ensure the accuracy of semantic matching.

[0081] 3. Global information integration: Integrate cross-modal association results with global context such as "Patient ID: P20240901" in the medical report header, ensure that local fields such as "blood sugar value" are correctly attributed, and avoid "field value mismatch" situations, such as incorrectly assigning a patient's blood sugar value to another patient.

[0082] Taking medical test sheet image analysis as an example, visual feature extraction identifies "the 4th row, 3rd column cell" containing the number "120 / 80 mmHg", and text semantic analysis identifies the "systolic / diastolic pressure" field. Cross-modal association through similarity calculation binds the two as "blood pressure value: 120 / 80 mmHg", while associating the "patient ID: P20240901" header, forming a complete semantic pair of "patient ID - blood pressure value".

[0083] S3, Zero-shot structured generation. Based on dynamic prompt engineering, guide multi-modal large models such as GPT-4V and LLaVA to output structured data according to the preset Schema, without the need to fine-tune the model for specific scenarios, the process is as follows:

[0084] S31, Target Schema analysis and prompt template generation.

[0085] 1. Schema parsing: Read the user's preset structured output format, such as JSON Schema, database table structure, extract the field name, data type, constraint condition, such as "amount: float, need > 0" "detection time: ISO8601 format", and specify the output specification of structured data.

[0086] 2. Domain adaptation prompt generation: According to the application scenarios of finance, medical treatment, retail, etc., generate special prompt templates, which include "role definition - task description - format constraint - rule description" four parts. Take the medical scene as an example, its prompt template is expressed in python language as follows:

[0087] PROMPT_TEMPLATE = f"""

[0088] You are a medical data structured conversion expert, and you need to handle multi-modal input (laboratory sheet image + text report).

[0089] Task: Output structured data in the following JSON format, with the following field constraints:

[0090] {{

[0091] "patient ID": "string, required, format 'P + 8 digits'",

[0092] "test date": "ISO 8601 format (YYYY - MM - DDTHH:MM:SS), required",

[0093] "blood pressure value": "string (value + unit, such as '120 / 80 mmHg'), required",

[0094] "blood sugar value": "float (unit: mmol / L), optional, output null if not present"

[0095] }}

[0096] Rules:

[0097] 1. Field values must correspond to patient ID to avoid cross-patient mismatch;

[0098] 2. If the field value cannot be identified from the input, output null;

[0099] 3. Numerical fields need to be checked for reasonableness (such as blood sugar value range 3.9 - 6.1 mmol / L, exceed 'to be reviewed').

[0100] """

[0101] S32, prompt word optimization and multi-modal feature splicing.

[0102] 1. Prompt optimization: Through A / B testing, compare the generation accuracy of different prompt structures such as field order, constraint description detail level, and adjust the prompt word weight through reinforcement learning such as PPO algorithm, for example, increase the weight of "numerical value rationality verification" by 20%, to reduce the error output and improve the guiding effect of prompt words on large models.

[0103] 2. Feature splicing: splice the pre-processed multi-modal features including visual block embedding, text word embedding, cross-modal association results and optimized prompts to form the input sequence of the large model, ensuring that the model can simultaneously obtain "semantic information" and "format constraints" to provide comprehensive basis for accurate structured data generation.

[0104] S33, Large model structured generation and candidate screening.

[0105] 1. Autoregressive generation: The multi-modal large model fills in the values field by field based on the input sequence using autoregressive method to generate the initial structured data such as JSON format data.

[0106] 2. Candidate generation and screening: Use Beam Search to generate Top-3 candidate structured results, calculate the confidence of each candidate result, for example, the model's confidence for "blood pressure value: 120 / 80 mmHg" is 0.95, and select the candidate result with the highest confidence as the initial output. The following is an example:

[0107] {

[0108] "patient ID": "P20240901",

[0109] "test date": "2024-09-01T08:30:00",

[0110] "blood pressure value": "120 / 80 mmHg",

[0111] "blood glucose value": 5.6

[0112] }

[0113] S4, Adaptive post-processing. Through "error correction module" and "normalization module", solve the problems of "abnormal values" and "format inconsistency" in the large model generation results, ensure that the structured data meets the business standards, the specific operation is as follows:

[0114] S41, Error correction module: anomaly detection and filtering.

[0115] 1. Rule engine verification: Configure industry-specific verification rules based on Drools rule engine, automatically detect abnormal fields. Take the financial invoice scenario and medical scenario as an example, the specific rules are as follows:

[0116] Financial invoice amount verification rule

[0117] rule "Finance_Invoice_Amount_Check"

[0118] when

[0119] $data : StructuredData(invoiceAmount <= 0 || invoiceAmount >1000000)

[0120] then

[0121] $data.setInvoiceAmount(null);

[0122] $data.addRemark("Invoice amount is abnormal (must be > 0 and <= 1 million), marked for manual review");

[0123] end

[0124] Medical blood pressure value verification rule

[0125] rule "Medical_BP_Check"

[0126] when

[0127] $data : StructuredData(bloodPressure!= null &&!bloodPressure.matches("\\d+ / \\d+ mmHg"))

[0128] then

[0129] $data.setBloodPressure(null);

[0130] $data.addRemark("Blood pressure value format error (must be 'value / value mmHg'), marked for manual review");

[0131] end

[0132] 2. Confidence filtering: Set a confidence threshold, such as 0.8. For fields with model output confidence below this threshold, such as "detection date" with a confidence of 0.7, automatically mark them as "to be manually reviewed" to prevent low-confidence data from entering the business system and affecting business decisions.

[0133] S42, normalization module: unify format and unit.

[0134] 1. Format standardization.

[0135] Time format: use regular expressions to convert different formats of time such as "2024-09-01 8:30" and "2024-09-01" into ISO 8601 format, i.e. "2024-09-01T08:30:00", to ensure consistency of time data format.

[0136] Value format: convert different representations of numerical values such as "1 million 5 thousand 8 hundred yuan" and "15,800 yuan" into floating-point numbers "15800.0", and keep two decimal places, such as converting "5.6" to "5.60", to achieve standardization of numerical value format.

[0137] 2. Unit unification.

[0138] Weight unit: convert different weight units such as "5kg" and "8 pounds" to "kilograms", and the converted values are "5 kilograms" and "3.63 kilograms" respectively, to ensure the uniformity of weight data units.

[0139] Amount unit: convert different amount units such as "5 million yuan" and "100 dollars" to "yuan" according to the exchange rate on the same day, and the converted values are "50000.0 yuan" and "730.0 yuan" respectively, to achieve consistency of amount data units.

[0140] S43, manual review interface. For fields marked as "to be reviewed", such as abnormal amounts and low-confidence dates, the system provides a web-based visualization review interface. This interface will display "original multi-modal data - abnormal fields - candidate values", and staff can manually correct on the interface. After correction, the system will trigger the post-processing process again to ensure the accuracy of the final output data.

[0141] S5, output and storage. The standardized structured data after post-processing is output and stored according to user needs, while retaining the link of the original heterogeneous data, forming a traceable closed loop of "structured data - original data".

[0142] S51, structured data output. The system supports multiple formats of output to adapt to the needs of different business systems:

[0143] 1. File format: JSON format can be output, suitable for API interface transmission; XML format can be output, suitable for enterprise-level system integration; CSV format can be output, suitable for Excel analysis, meeting the data usage needs of different business scenarios.

[0144] 2. Database format: Automatically generate SQL statements, including CREATE TABLE and INSERT statements, to write structured data into a relational database, and create indexes for relevant fields based on field types, such as indexes for "patient ID" and "invoice number" to improve database query efficiency.

[0145] S52, data storage optimization.

[0146] 1. Structured data storage: When writing structured data into a relational database, use a table splitting strategy, such as storing medical data by "test date" to avoid slow query speed caused by large data volume in a single data table and improve data query efficiency.

[0147] 2. Raw data storage: Store pre-processed raw heterogeneous data, including images, audio / video, and text, in object storage services such as MinIO and AWS S3, while retaining the storage path and metadata of the original data in the structured data, such as "original invoice image path: / minio / invoice / 20240901 / INV - 20240901.jpg", to provide support for data traceability.

[0148] 3. Log recording: Detailed log information of the entire conversion process is recorded, such as "2024 - 09 - 01 09:00: Invoice INV - 20240901 conversion completed, time 2.5 seconds, no abnormal fields", which facilitates subsequent problem troubleshooting and process monitoring, and ensures the stable operation of the conversion process.

[0149] S53, output verification. The system automatically checks the integrity and consistency of the output data, such as checking whether the blood pressure value and blood glucose value of the record with patient ID P20240901 in the database are consistent with the JSON output. If inconsistent data is found, the system will trigger a retry mechanism to re-execute steps three to five to ensure the reliability of the output data and provide high-quality data support for business applications.

[0150] The embodiment has the advantages that through the synergistic effect of the above steps, the structured data has high accuracy in the scenes of financial bill processing, medical report analysis, industrial data conversion, etc., is significantly improved compared with the traditional OCR+rule template method, and supports zero sample adaptation to new scenes, such as adding retail commodity label conversion, greatly reduces the model migration cost, and provides an efficient and reliable solution for heterogeneous data conversion in various industries.

[0151] Embodiment two

[0152] The embodiment of the application is a heterogeneous data conversion system based on a multi-modal large model, a multi-layer modular system of an input adaptation layer-a multi-modal analysis layer-a structured generation layer-a self-adaptive post-processing layer-an output storage layer is constructed, the responsibilities of each layer are clear and cooperate with each other, and image, text, audio and video heterogeneous data can be automatically converted into standardized structured data, the scalability and business adaptability are taken into account, and the specific functions of each module are as follows:

[0153] I. Input adaptation layer. As the data entry of the system, the core is to compatible multi-source data and optimize quality, to pave the way for the subsequent links, including two units:

[0154] 1. Format adaptation unit: supports mainstream formats such as image JPEG, PNG, TIFF; text PDF, DOCX, TXT, including editable and scanned versions, audio WAV / MP3, video MP4 / AVI, etc. When encountering non-standard formats such as special compression TIFF and encrypted PDF, it is automatically converted into a standard format that can be processed by the system to ensure compatibility.

[0155] 2. Type-based preprocessing unit: different data types are processed differently. Image type removes noise, optimizes blurred images, preliminarily locates text areas and identifies content; text type extracts content and format information, filters redundant characters, corrects OCR errors, and marks key entities such as financial "invoice number" and medical "patient ID"; audio and video types convert audio to text with timestamps and extract tone features, and take key frames at 1 frame per second to establish a "frame-text-time" mapping.

[0156] II. Multi-modal analysis layer. Responsible for breaking the single-modal limitation, establishing cross-modal semantic association, and outputting semantic-aligned feature pairs, including three units:

[0157] 1. Visual feature extraction unit: uses ViT model and Mask R-CNN tool to analyze images. First, identify structured elements such as tables and charts to determine their positions and categories; then, split the image into pixel blocks to generate visual features, and extract additional cell position information for table areas to provide a basis for text association.

[0158] 2. Text semantic analysis unit: Understand the text based on BERT and BiLSTM models. Convert the pre-processed text into semantic features, strengthen the relevance of information in long text, and identify core fields using industry-specific tools, such as "invoice number: INV-20240901", where "invoice number" is the field name and "INV-20240901" is the field value.

[0159] 3. Cross-modal association unit: Associate multi-modal data through Transformer cross-attention mechanism. Calculate the similarity between visual features and text features, generate an association matrix, and preferentially bind high-similarity features such as "blood pressure" text and "120 / 80 mmHg" cell, and combine global information such as medical report "patient ID" to avoid field mismatch.

[0160] Three, structured generation layer. Rely on zero-shot learning to let multi-modal large models output structured data in pre-set formats without scene fine-tuning, including three units:

[0161] 1. Schema analysis unit: Read user-set output formats such as JSON Schema and database table structure, extract field names, data types, and constraint conditions, and generate machine-readable metadata to provide the basis for prompt templates.

[0162] 2. Prompt engineering unit: Generate prompt templates for financial, medical, and other scenarios, including roles, tasks, formats, and rules. Optimize the structure of prompt words through testing, and then concatenate the semantic alignment features of the multi-modal analysis layer with the template to form the input content of the large model.

[0163] 3. Large model generation unit: Call multi-modal large models such as GPT-4V and LLaVA to generate initial structured data based on input. Use beam search to generate three candidate results, and select the optimal one as the initial output such as JSON format, while recording the confidence of each field for subsequent use.

[0164] Four, adaptive post-processing layer. Correct the problems of the large model output to ensure that the data meets business standards, including three units:

[0165] 1. Error correction unit: Use Drools rule engine to check data, such as detecting "invoice amount ≤0 or >100 million" in financial scenarios and checking "blood pressure value format is 'value / value mmHg'" in medical scenarios. Mark "to be reviewed" and set to null for abnormal fields, and add notes; At the same time, set the confidence threshold, such as 0.8, and mark "to be reviewed" for fields below the threshold.

[0166] 2. Normalization unit: unify data format and unit. Time is unified to ISO 8601 format, such as "2024-09-01T08:30:00"; numerical values are converted to floating-point numbers "15800.0" with two decimal places; units are converted to "kg" from "pounds" and "yuan" from "dollars" according to the exchange rate.

[0167] 3. Artificial review unit: provides a web-based visualization interface to display "raw data - abnormal field - candidate value" and supports manual correction. After correction, the post-processing process is automatically triggered again, and operation logs are recorded to ensure data traceability.

[0168] Five, output storage layer. System data export, responsible for output, storage and traceability, forming a closed loop, including three units:

[0169] 1. Format output unit: supports multiple format output. File formats can output JSON for API transmission, XML for enterprise system integration, and CSV for Excel analysis; database format automatically generates SQL statements to write data into MySQL, PostgreSQL, etc. and creates indexes for key fields such as "patient ID" to improve query efficiency.

[0170] 2. Storage optimization unit: hierarchical storage of data. Structured data is stored in tables according to scenarios, such as medical data stored in tables according to "test date" to avoid excessive data in a single table; raw data is stored in object storage services such as MinIO and AWS S3; structured data retains the original data storage path; conversion logs such as "2024-09-01 09:00: Invoice INV-20240901 conversion completed, time 2.5 seconds, no exceptions" are recorded to facilitate problem troubleshooting.

[0171] 3. Output verification unit: automatically checks data. Consistency between different formats is checked, such as whether the values of the same field in JSON and the database are consistent; the association between structured data and raw data is checked, such as whether the invoice amount in the database matches the raw image amount; if not, the "structured generation layer - output storage layer" process is re-executed to ensure data reliability.

[0172] This system cooperates through multiple modules to ensure the accuracy of structured data in financial bill processing and medical report analysis while significantly reducing model migration costs. The modules are loosely coupled, making it easy to extend functionality in the future.

[0173] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several improvements and refinements can be made without departing from the technical principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A heterogeneous data conversion method based on a multimodal large model, characterized in that, Includes the following steps: S1. Data Input and Preprocessing: For multi-source heterogeneous input data, format regularization is performed, and preliminary information extraction is carried out based on data type differences; the multi-source heterogeneous input data includes image data, text data, and audio / video data; S2. Multimodal Joint Analysis: Through visual feature extraction, text semantic analysis, and cross-modal association processes, a semantic mapping between data from different modalities is established. S3. Zero-sample structured generation: Based on dynamic prompting engineering, it guides multimodal large models to output structured data according to a preset schema; S4. Adaptive Post-processing: By using error correction and normalization functions, it solves the problems of outliers and inconsistent formats in the generated results of large models, so that structured data conforms to business standards. S5. Output and Storage: Output and store structured data that meets business standards according to user needs, while retaining the original heterogeneous data link to form a traceable closed loop from structured data to original data; Step S2 includes: S21. Visual Feature Extraction: Mask R-CNN is used to identify structured elements in the image and output bounding box coordinates and category labels. The image is then divided into pixel blocks, and the block sequence features are generated by mapping them to the dimensional embedding space through the linear projection layer of ViT. Cell position features are extracted for table areas. The structured elements include tables, charts, and icons. S22. Text Semantic Parsing: Input the preprocessed text into the BERT model to generate word-level embedding vectors, strengthen the semantic association of long texts through the BiLSTM model, and combine the industry-specific NER model to identify the core fields in the text; S23. Cross-modal association: Using the Transformer cross-attention mechanism, the cosine similarity between visual block embeddings and text word embeddings is calculated to generate a "visual-text association matrix". In the cross-attention layer, the multi-head attention mechanism is used to strengthen the association of high similarity feature pairs, and the cross-modal association results are fused with the global context.

2. The heterogeneous data conversion method based on a multimodal large model according to claim 1, characterized in that, Step S1 includes: S11. Access to multi-source heterogeneous input data formats, data type classification and adaptation; the image class includes JPEG, PNG, and TIFF formats; the text class includes PDF, DOCX, and TXT formats; the audio and video class includes WAV, MP3, MP4, and AVI formats; S12. Data preprocessing by type: For image data, format unification, quality optimization, and preliminary text localization are performed. Format unification converts non-standard formats to JPEG format. Quality optimization uses Gaussian filtering for noise reduction and adaptive threshold binarization to process blurred images. Preliminary text localization uses the Tesseract OCR tool to detect text regions and output relevant information. For text data, format parsing, noise filtering, and preliminary entity labeling are performed. Format parsing uses Apache PDFBox and Python-docx tools to extract text content and format information. Noise filtering removes redundant characters and corrects OCR-recognized typos. Preliminary entity labeling uses jieba word segmentation combined with a custom industry dictionary to label key entities. For audio and video data, audio transcription, video keyframe extraction, and "frame-text-time" mapping are completed. Audio processing uses the Whisper model to transcribe text and retain timestamps while extracting audio features. Video processing extracts keyframes by frequency and performs image preprocessing, binding the transcribed audio text with the corresponding frame timestamps.

3. The heterogeneous data conversion method based on a multimodal large model according to claim 1, characterized in that, Step S3 includes: S31. Target Schema Parsing and Prompt Template Generation: Read the user-preset structured output format, extract field names, data types, and constraints, and generate a dedicated prompt template containing "role definition - task description - format constraints - rule description" according to the application scenario; the structured output format includes JSON Schema and database table structure; S32. Prompt word optimization and multimodal feature concatenation: By testing and comparing the generation accuracy under different prompt structures, the prompt word weights are adjusted in combination with reinforcement learning algorithms, and the preprocessed multimodal features and optimized prompt words are concatenated to form a large model input sequence; S33. Large Model Structure Generation and Candidate Selection: The multimodal large model generates initial structured data by filling in numerical values ​​field by field according to the input sequence using an autoregressive method, generates candidate structured results using bundle search, calculates the confidence score of each candidate result, and selects the candidate result with the highest confidence score as the initial output; the multimodal large model is GPT-4V or LLaVA.

4. The heterogeneous data conversion method based on a multimodal large model according to claim 1, characterized in that, Step S4 includes: S41. Configure industry-specific validation rules based on the Drools rule engine to automatically detect abnormal fields, set a confidence threshold, and mark fields whose confidence level in the model output is lower than the confidence threshold as "awaiting manual review"; S42. Standardize the format, convert different time formats to ISO 8601 format, convert different numerical representations to floating-point numbers and retain a fixed number of decimal places; standardize the units; S43. Provides a manual review interface, which displays "raw multimodal data - abnormal fields - candidate values" through a web-based visual review interface, allowing staff to make manual corrections.

5. The heterogeneous data conversion method based on a multimodal large model according to claim 1, characterized in that, Step S5 includes: S51. Output structured data; the structured data includes structured file format and database format; the structured file format includes JSON format, XML format, and CSV format; for database format output, generate SQL statements, write the structured data into a relational database, and create indexes for relevant fields; S52. Data storage optimization: When writing structured data to a relational database, a table partitioning strategy is adopted; the raw heterogeneous data before preprocessing is stored in an object storage service, and the storage path and metadata of the raw data are retained in the structured data; detailed log information of the entire transformation process is recorded. S53. Output Verification: The system verifies the integrity and consistency of the output data. If inconsistency is found, a retry mechanism is triggered, and the process from step S3 to step S5 is re-executed.

6. A heterogeneous data conversion system based on a multimodal large model, characterized in that, It includes an input adaptation layer, a multimodal parsing layer, a structured generation layer, an adaptive post-processing layer, and an output storage layer: (1) Input adaptation layer: including format adaptation unit and type preprocessing unit; the format adaptation unit supports the access of mainstream formats of multi-source heterogeneous data and converts non-standard formats; the type preprocessing unit performs differentiated preprocessing operations for different types of data, optimizes data quality and extracts preliminary information; (2) Multimodal parsing layer: It consists of a visual feature extraction unit, a text semantic parsing unit and a cross-modal association unit. The visual feature extraction unit parses the structured elements and visual features of the image. The text semantic parsing unit understands the text semantics and identifies key entities. The cross-modal association unit establishes cross-modal semantic associations. The cross-modal association unit generates an association matrix by calculating the cosine similarity between the visual block embedding and the text word embedding. In the cross attention layer, the multi-head attention mechanism is used to strengthen the association of high similarity feature pairs. The association results are integrated with the global context. (3) Structured generation layer: including schema parsing unit, prompting engineering unit and large model generation unit. The schema parsing unit extracts preset output schema information, the prompting engineering unit generates and optimizes prompt templates, and the large model generation unit calls multimodal large model to generate and filter structured data. (4) Adaptive post-processing layer: includes an error correction unit, a normalization unit and a manual review unit. The error correction unit verifies and marks abnormal and low-confidence data. The normalization unit unifies the data format and unit. The manual review unit provides an interface to support manual correction of data. (5) Output storage layer: It consists of a format output unit, a storage optimization unit and an output verification unit. The format output unit outputs multi-format data, the storage optimization unit stores data in layers and records logs, and the output verification unit verifies data consistency and triggers a retry mechanism.

7. The heterogeneous data conversion system based on a multimodal large model according to claim 6, characterized in that, In the categorized preprocessing unit, image preprocessing uses Gaussian filtering for noise reduction and adaptive threshold binarization to optimize the image, and Tesseract OCR tool to locate text regions; text preprocessing uses Apache PDFBox and Python-docx tools to extract text information, and jieba word segmentation combined with industry dictionary to mark key entities; audio and video preprocessing uses Whisper model to transcribe audio and extract intonation features, and extracts video keyframes by frequency.

8. The heterogeneous data conversion system based on a multimodal large model according to claim 6, characterized in that, The storage optimization unit stores structured data in separate tables according to the scenario, and the format output unit creates indexes for key fields in the database.

Images