Method for processing multi-modal mixed data and enhancing model generation effect

Abstract

The invention discloses a method for processing multi-modal mixed data and enhancing a model generation effect, and belongs to the technical field of data process.The method constructs a multi-modal analysis tool chain, performs analysis and feature extraction on multi-modal data through the multi-modal analysis tool chain, solves the problem that data processing is difficult, and improves the data processing efficiency. Compared with the prior art, the method has the advantages that the processing time of a single piece of data is shortened, the data processing cost is reduced, the hardware investment is reduced by 80% compared with a workstation cluster required by a manual annotation team, mutual association of the multi-modal data is realized by establishing the multi-modal data association relationship and the graph database, the problem of modal splitting is solved, and the data processing efficiency is improved. The technical effects of text image searching and text table searching can be achieved through the incidence relation, the semantic barrier of text discrete symbols and image continuous representation is broken through, and image-text content correlation response is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a method for processing multimodal mixed data and enhancing model generation effect. Background Art

[0002] Big AI models refer to a class of AI models with a large number of parameters, constructed using artificial neural networks. They are typically pre-trained on massive amounts of data through self-supervised or semi-supervised learning, and their performance and capabilities are further optimized through methods such as instruction fine-tuning and human alignment. Big models are characterized by a large number of parameters, extensive training data, and extensive computing resources. They possess the ability to solve general tasks, follow human instructions, and perform complex reasoning. Major categories of big AI models include big language models, big vision models, big multimodal models, and big basic science models. Currently, big models have been widely used in a variety of fields, including search engines, intelligent agents, related vertical industries, and basic science, driving the intelligent development of various industries.

[0003] However, the following problems exist when large models are implemented in vertical fields: 1. Difficulty in data processing: The core challenge in implementing large models in vertical fields lies in data processing. With the explosive growth of multimodal, unstructured data (including text, images, video, tables, and audio), data size and complexity have surged, creating an imbalance between annotation efficiency and cost. Fields like healthcare and the legal system require expert annotation, with an average daily efficiency of only 20-50 items per person, and annotation costs accounting for 60%-80% of the total model development cost. In industrial scenarios, the lack of multimodal data annotation standards, such as the correspondence between equipment vibration signals and fault text, leads to quality fluctuations of up to 15%-30%. Furthermore, technical barriers exist in specialized fields, such as medical symptom-pathology causal chains and judicial law-case matching logic, which exceed the capture capabilities of general corpora. This results in a 20%-40% decrease in semantic parsing accuracy compared to general scenarios.

[0004] 2. Modal fragmentation: Cross-modal information is difficult to associate, and there is a dilemma in the coordination of multimodal data. Traditional retrieval technology faces significant bottlenecks in dealing with multimodal data association. Processing mechanisms based on a single modality, such as text keyword matching or image feature extraction, are difficult to break through the semantic barriers between modalities. Accurate mapping cannot be established under the traditional technical framework, and cross-modal semantic consistency errors often reach more than 30%. The root cause lies in the structural differences between the discrete symbolic expression of text and the continuous spatial representation of images, which makes the similarity calculation of the general retrieval model unable to capture deep semantic associations.

[0005] 3. Insufficient generation capabilities: Large language models are limited to text output and cannot generate visual modality content, such as three-dimensional models of industrial equipment and medical image reconstruction, resulting in limited application scenarios. Existing models lack cross-modal dynamic mapping capabilities, and the generated content often has logical discontinuities between images and text. For example, the image description in a medical report is disconnected from the pathological analysis, and the semantic consistency is 40%-60% lower than that of manual creation.

[0006] These situations not only increase the cost of data cleaning and organization, but also restrict the domain adaptation and scenario-based application effects of large models from the source of data. Summary of the Invention

[0007] In order to solve the problems raised in the above background technology, the present invention provides a method for processing multimodal mixed data and enhancing the model generation effect, so as to solve the problems of difficult data processing, modal fragmentation and insufficient generation capability in traditional technical solutions.

[0008] To achieve the above object, the present invention provides the following technical solutions: A method for processing multimodal mixed data and enhancing model generation effect, comprising the following steps: S1: Build a distributed data storage layer and a multimodal parsing toolchain. The distributed data storage layer includes a vector storage cluster and an object storage cluster. The multimodal parsing toolchain parses each source document, extracts multimodal metadata from each source document, including text, images, and tables, and stores them in the object storage cluster. The multimodal parsing toolchain semantically segments text metadata into paragraphs, generates descriptive text for image metadata, and extracts structured features from table metadata. S2: Generate high-density representation vectors for text paragraphs and build a multi-layer index structure based on the high-density representation vectors through vector storage clusters; S3: Assign a globally unique UUID to each data entity, including text paragraphs, description text, and structured features; S4: Generate associations for data entities of text, image, and table metadata in the same source document, and use regular expressions to match reference tags to build a graph database. Each data entity is set as a node, and the associations between data entities are set as edges. A hybrid index is built on the graph database. S5: Convert user questions into high-dimensional semantic vectors. Based on the multi-layer index structure of the vector storage cluster, the nearest neighbor algorithm is used to recall the top-K matching candidate text paragraphs. Combined with the filtering operation including relevance scoring and logical conflict detection, the filtered text paragraphs are output. S6: Based on the filtered text paragraphs, by parsing their corresponding reference tags, the matching associated UUID objects are found based on regular expressions and output. Then, by training the multimodal alignment model, the image with a threshold similarity to the filtered text paragraphs is calculated by describing the text and output. By calling pre-stored materials and preset graphic and text mixed templates, the output UUID objects and images are inserted to generate the final response.

[0009] Preferably, the vector storage cluster is an elastically scalable vector storage cluster built using Milvus, and the object storage cluster is a distributed object storage cluster built based on Minio.

[0010] Preferably, the multimodal parsing tool chain includes the gte-Qwen2-7B model, the DeepSeek Janus Pro image understanding model and the Pandas library, wherein the gte-Qwen2-7B model is used to semantically segment the text metadata to generate text paragraphs, the DeepSeek Janus Pro image understanding model is used to generate descriptive text for the image metadata, and the Pandas library is used to perform structured feature extraction on the table metadata.

[0011] Preferably, the high-density representation vector in S2 is generated through an Embedding model.

[0012] Preferably, the UUID generation and allocation method is: using the RFC4122 standard, combining timestamps and random numbers to generate a globally unique UUID, allocating globally unique UUIDs to all data entities through a shard generator, and embedding the source system identifier, version number, and creation timestamp in the basic attributes of the UUID.

[0013] Preferably, the screening operation in S5 is specifically as follows: calling the cross-encoder model to score the relevance between the user question and each candidate text paragraph, dynamically adjusting the sorting priority including relevance score, document authority and update time, outputting the Top-N refined ranking results, configuring regular expressions and knowledge graphs to detect and verify the refined ranking results, and then using rule chains in combination with confidence thresholds to detect contradictory statements, and finally retaining the Top-M text paragraphs.

[0014] Preferably, it also includes S7: verifying the logical consistency of the graphics and text of the final response through the GNN model, and continuing to output the final response if consistent, and triggering re-retrieval and returning to S5 if inconsistent.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This application constructs a multimodal parsing tool chain, which parses and extracts features from multimodal data through the multimodal parsing tool chain, solving the problem of difficult data processing, shortening the processing time of a single piece of data, and reducing data processing costs. Compared with the workstation cluster required by the manual annotation team, the hardware investment is reduced by 80%. This application also realizes the mutual correlation of multimodal data by establishing multimodal data association relationships and graph databases, solving the problem of modal fragmentation, and through the association relationships, the technical effects of text search and text search can be achieved, breaking through the semantic barriers between discrete text symbols and continuous image representation, and realizing the association response of graphic and text content. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of the process of establishing the text-image-table association relationship for this application; Figure 2 Schematic diagram of the process for generating answers for this application. DETAILED DESCRIPTION

[0017] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] Example 1 A method to process multimodal mixed data and enhance the model generation effect, such as Figure 1 、 Figure 2 As shown, the following steps are included: S1: Build a distributed data storage layer and a multimodal parsing toolchain. The distributed data storage layer includes a vector storage cluster and an object storage cluster. The multimodal parsing toolchain parses each source document, extracts multimodal metadata from each source document, including text, images, and tables, and stores them in the object storage cluster. The multimodal parsing toolchain semantically segments text metadata into paragraphs, generates descriptive text for image metadata, and extracts structured features from table metadata. Among them, Milvus is used to build an elastic and scalable vector storage cluster that supports dynamic node expansion. The sharded storage mechanism enables efficient management of massive vector indexes, improving retrieval efficiency by over 300% compared to single-node PG vectors. Computing resources can be dynamically adjusted based on data scale, supporting petabyte-level data processing needs. A distributed object storage cluster is built based on Minio, and a multi-copy storage strategy is designed to ensure data security. A hierarchical storage architecture supports tiered management of hot / warm / cold data, reducing storage costs by over 40%. A standard "S3 interface" is provided. The multimodal parsing toolchain integrates professional document parsing tools such as MinerU and MegaParse, and combines the Umi-OCR image recognition engine and the DeepSeek Janus Pro multimodal model to build a complete parsing matrix. This supports automated processing of over 20 file formats and enables accurate extraction of elements such as text, images, and tables. The multimodal parsing toolchain uses the gte-Qwen2-7B model to semantically segment text metadata, generates graphic descriptions of image metadata based on the DeepSeek Janus Pro image understanding model, uses Pandas to extract structured features from table metadata, and finally applies bge-reranker-v2-m3 to improve information relevance. S2: Generate high-density representation vectors for text paragraphs through the Embedding model, and build a multi-layer index structure based on the high-density representation vectors through vector storage clusters; S3: Assigns a globally unique UUID to each data entity, including text paragraphs, descriptive text, and structured features. Specifically, it uses the RFC4122 standard, combining timestamps and random numbers to generate UUIDs to ensure global uniqueness. A sharding generator enables conflict-free UUID allocation in a distributed environment, supporting millions of entity registrations per second. Globally unique UUIDs are assigned to all data entities, such as text paragraphs, image files, and table datasets (at row and column level granularity). The source system identifier, version number, and creation timestamp are embedded in the basic UUID attributes to support multi-version traceability. S4: Generate associations for data entities of text, image, and table metadata in the same source document, and use regular expressions to match reference tags to build a graph database. Each data entity is set as a node, and node attributes include UUID, type, storage path, etc., as shown in Table 1. Table 1 is a schematic table of nodes. The associations between data entities are set as edges, and a hybrid index is built on the graph database. Table 1

[0019] This application also supports dynamic relationship updates and maintenance, including an incremental update mechanism: monitoring metadata change events through Change Data Capture (CDC) to trigger the graph structure update pipeline; transaction consistency: using graph database ACID transactions to ensure the atomicity of association operations, combined with asynchronous batch processing to improve large-scale update throughput; association quality monitoring: defining integrity indicators and consistency rules, and maintaining graph health through scheduled inspections and abnormal alarm mechanisms.

[0020] S5: Build an intelligent question-answering engine using the LangChain framework, using pre-trained language models (such as Alibaba-NLP / gte-Qwen2-7B-instruct) to convert user questions into high-dimensional semantic vectors (generating 3584-dimensional vectors), build a distributed vector index based on Milvus, and quickly recall the top-K candidate documents (such as K=200) through the approximate nearest neighbor (ANN) algorithm. Support sharded storage and dynamic expansion to meet millisecond-level response requirements under PB-level data scale. Then, perform preliminary result integration, retain high-recall but potentially redundant content, and output candidate documents. The system generates a list of answers and appends data such as the document source and timestamp. It then calls the cross-encoder model to score the relevance between the user question and each candidate answer. It dynamically adjusts the sorting priority based on document authority and update time, outputs the Top-N refined ranking results, and then configures regular expressions to detect key entities (such as the date format \d{4}-\d{2}-\d{2}) and verify factual relationships through the knowledge graph (such as "Beijing is the capital of China" through graph verification). Finally, it uses rule chains to detect contradictory statements and, combined with a confidence threshold, retains the Top-M high-quality answers (for example, M=5) for use in the generation phase.

[0021] The intelligent question-answering engine of this application also supports multi-instance load balancing of large models such as DeepSeek, including: Request distribution strategy: implement weighted polling or minimum connection algorithm based on the real-time load of the model instance (such as GPU utilization, response delay); context-aware routing: select the model instance adapted to the domain according to the problem type, and dynamically match it through preset tags; health check mechanism: poll the HTTP health interface of the model instance every 5 seconds, and the abnormal instance is automatically marked as "unavailable" and triggers an alarm; elastic scaling: automatically start the container instance based on the QPS (requests per second) threshold, and support scaling in seconds; dynamic prompt word engineering optimization: automatically select the prompt template according to the query type, support dynamic variable injection (such as {current_date} replaced with today's date), through Session The ID tracks historical context and appends a summary of the previous conversation to the prompt word. The resulting credibility assessment mechanism includes the following: ① LLM self-assessment: Invokes the self-developed BART classification model to determine the factuality of the generated content (outputs a [True / False] label); ② Rule verification: Detects sensitive words and compliance; ③ Voting mechanism: Multiple models vote on generated results; ④ Confidence score calculation: Generates a confidence score based on comprehensive semantic similarity, factual consistency, and source authority. A confidence score below 60% triggers manual review.

[0022] S6: Based on the filtered text paragraphs, the corresponding reference tags are parsed, and matching UUID objects are found based on regular expressions and output. A multimodal alignment model is then trained to calculate images that meet a threshold similarity threshold with the filtered text paragraphs using the description text. The LangChain framework then calls Tools or FunctionCall to pull pre-stored assets and pre-set images and text from object storage (Minio), combining the output content to generate the final response. Chunked Transfer Encoding is used to achieve progressive loading of images and text, achieving a first-screen response time of less than 1 second.

[0023] S7: Verify the logical consistency of images and texts through the GNN model, and trigger re-retrieval when there is inconsistency.

[0024] In this embodiment, the present application also develops a distributed task scheduling engine with the following technical solutions: Intelligent pipeline orchestration: Based on task dependencies, a dynamic orchestration engine enables collaborative multi-task scheduling. This utilizes the Kubernetes workflow framework (defined task nodes, supporting conditional branching, loop execution, and parallel scheduling). Combined with priority queues and task label matching mechanisms, this enables intelligent routing of tasks across clusters. For example, in financial trading scenarios, risk control calculation tasks take precedence over data cleaning tasks to ensure critical path timeliness.

[0025] Fault recovery and breakpoint resumption mechanisms: ① Health Check: Heartbeat detection (such as Consul HealthCheck) monitors the status of Worker nodes in real time, automatically isolates abnormal nodes, and triggers task migration. ② State Snapshot: Checkpointing technology is used to persist intermediate task states and support resuming execution from the most recent valid breakpoint. ③ Retry Strategy: Tiered retry times (e.g., retry after 5 seconds for the first attempt, followed by exponential backoff for subsequent attempts) are combined with circuit breakers to prevent cascading failures.

[0026] Dynamic Resource Allocation Algorithm: ① Hybrid Scheduling Strategy: This combines priority scheduling (high-priority tasks preempt resources) with a load balancing algorithm (based on weighted round-robin resource utilization), implemented through the Kubernetes default scheduler and custom schedulers such as Volcano and Queue. ② Elastic Scaling: Based on CPU and memory utilization metrics collected by Prometheus, KEDA (an event-driven autoscaler for Kubernetes) triggers the Horizontal Pod Autoscaler to dynamically adjust compute instances, supporting burst traffic handling. ③ GPU Resource Sharing: Physical GPU sharding is achieved through technologies such as NVIDIA MIG and HAMi, dynamically allocating computing resources based on task demand.

[0027] Real-time monitoring and alerting system: Prometheus and Node Exporter are used for full-link monitoring, covering over 200 metrics, including task queue depth, latency percentile (P99), and node load. Multi-level thresholds (Warning / Critical) are configured using Grafana, and time series prediction (Prophet algorithm) is used to provide early warning of abnormal trends. Distributed tracing is implemented using Jaeger to quickly identify performance bottlenecks across service call chains.

[0028] A tiered circuit breaker strategy: triggers a rapid failure when the API error rate exceeds 30%. A node-level circuit breaker takes a node offline after three consecutive health check failures. A cluster-level circuit breaker automatically switches to a disaster recovery cluster in the event of a regional failure. After the circuit breaker, 5% of the traffic is periodically released to test service recovery, gradually restoring to full capacity.

[0029] This application also establishes a quality assurance system, specifically: Build a four-dimensional monitoring system: ① Data processing quality monitoring: By defining data quality rules, we scan and verify data at the table partition level to ensure data integrity. We use the Isolation Forest algorithm to identify outliers in numeric fields and, in conjunction with the rule engine, flag suspicious records. By building a data lineage graph, we can quickly locate the source of upstream dirty data. ② Model output compliance testing includes: multi-dimensional assessment (content security, fact-checking) and the rule engine: configuring regular expressions to filter PII (such as ID numbers and bank card numbers), combining the NER model to identify privacy leakage risks, and manual review: high-risk content is automatically transferred to the annotation platform, and violations are confirmed through a multi-person voting mechanism.

[0030] Real-time system performance analysis: Monitor container resource utilization at the infrastructure layer, collect microservice call chain performance data at the service layer, analyze front-end page loading performance at the application layer, and then predict bottlenecks: Based on historical performance data trained by the LSTM model, predict the CPU / memory usage peak in the next three hours.

[0031] Evaluate resource utilization efficiency across multiple metrics, including compute resources (GPU utilization, vCPU allocation ratio), storage resources (cost ratio of hot and cold data storage, such as based on MinIO lifecycle policies), network resources (bandwidth utilization / packet loss rate, through sampling analysis), and GPU resources (memory utilization).

[0032] This application has built a multimodal parsing tool chain, which solves the problem of difficult data processing, shortens the processing time of a single piece of data, and reduces the data processing cost. Compared with the workstation cluster required by the manual annotation team, the hardware investment is reduced by 80%. This application realizes the mutual correlation of multimodal data by establishing multimodal data association relationships and graph databases, solves the problem of modal fragmentation, and realizes the technical effects of text search and text search through association relationships, breaking through the semantic barriers between discrete text symbols and continuous image representation, and realizing the association response of graphic and text content.

Claims

1. A method for processing multimodal mixed data and enhancing model generation effect, characterized in that: The following steps are involved: S1: Build a distributed data storage layer and a multimodal parsing toolchain. The distributed data storage layer includes a vector storage cluster and an object storage cluster. The multimodal parsing toolchain parses each source document, extracts multimodal metadata from each source document, including text, images, and tables, and stores them in the object storage cluster. The multimodal parsing toolchain semantically segments text metadata into paragraphs, generates descriptive text for image metadata, and extracts structured features from table metadata. S2: Generate high-density representation vectors for text paragraphs and build a multi-layer index structure based on the high-density representation vectors through vector storage clusters; S3: Assign a globally unique UUID to each data entity, including text paragraphs, description text, and structured features; S4: Generate associations for data entities of text, image, and table metadata in the same source document, and use regular expressions to match reference tags to build a graph database. Each data entity is set as a node, and the associations between data entities are set as edges. A hybrid index is built on the graph database. S5: Convert user questions into high-dimensional semantic vectors. Based on the multi-layer index structure of the vector storage cluster, the nearest neighbor algorithm is used to recall the top-K matching candidate text paragraphs. Combined with the filtering operation including relevance scoring and logical conflict detection, the filtered text paragraphs are output. S6: Based on the filtered text paragraphs, by parsing their corresponding reference tags, the matching associated UUID objects are found based on regular expressions and output. Then, by training the multimodal alignment model, the image with a threshold similarity to the filtered text paragraphs is calculated by describing the text and output. By calling pre-stored materials and preset graphic and text mixed templates, the output UUID objects and images are inserted to generate the final response.

2. The method for processing multimodal mixed data and enhancing model generation effect according to claim 1, characterized in that: The vector storage cluster is an elastic and scalable vector storage cluster built using Milvus, and the object storage cluster is a distributed object storage cluster built based on Minio.

3. The method for processing multimodal mixed data and enhancing model generation effect according to claim 1, characterized in that: The multimodal parsing toolchain includes the gte-Qwen2-7B model, the DeepSeek Janus Pro image understanding model, and the Pandas library. The gte-Qwen2-7B model is used to semantically segment text metadata to generate text paragraphs, the DeepSeek Janus Pro image understanding model is used to generate descriptive text from image metadata, and the Pandas library is used to extract structured features from table metadata.

4. The method for processing multimodal mixed data and enhancing model generation effect according to claim 1, characterized in that: The high-density representation vector in S2 is generated through the Embedding model.

5. The method for processing multimodal mixed data and enhancing model generation effect according to claim 1, characterized in that: The UUID generation and allocation method is: using the RFC4122 standard, combining timestamps and random numbers to generate a globally unique UUID, allocating globally unique UUIDs to all data entities through a shard generator, and embedding the source system identifier, version number, and creation timestamp in the basic attributes of the UUID.

6. The method for processing multimodal mixed data and enhancing model generation effect according to claim 1, characterized in that: The specific filtering operations in S5 are as follows: calling the cross-encoder model to score the relevance between the user question and each candidate text paragraph, dynamically adjusting the sorting priority based on relevance score, document authority, and update time, outputting the Top-N refined ranking results, configuring regular expressions and knowledge graphs to detect and verify the refined ranking results, and then using rule chains in combination with confidence thresholds to detect contradictory statements, and finally retaining the Top-M text paragraphs.

7. The method for processing multimodal mixed data and enhancing model generation effect according to claim 1, characterized in that: It also includes S7: Verify the logical consistency of the final response image and text through the GNN model. If consistent, the final response continues to be output. If inconsistent, trigger re-retrieval and return to S5.

Images