Fine tuning data set generation method and device for multi-mode large model fine tuning, equipment and medium

Abstract

The embodiment of the invention provides a fine tuning data set generation method and device for fine tuning of a multi-modal large model, equipment and a medium, and the method comprises the steps: obtaining cross-modal data from original data of various file formats; based on the cross-modal data, constructing a three-dimensional domain knowledge graph including subject classification, terminologies and entity relationships; according to the three-dimensional domain knowledge graph, generating a question-answer pair and a thinking chain for the cross-modal data through an LLM model federated architecture; and performing semantic alignment on various modal data in the cross-modal data to obtain the fine adjustment data set. The scheme can improve the processing efficiency and improve the data quality.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and big data processing technology, and in particular to a method, apparatus, device and medium for generating fine-tuning datasets for fine-tuning multimodal large models. Background Technology

[0002] With the rapid development of artificial intelligence technology, fine-tuning large models has become an important means for various industries to achieve intelligent transformation. While numerous mature fine-tuning tools exist in the market, practice shows that the key factor determining the fine-tuning effect is not the algorithm itself, but rather the quality of the initial dataset construction. The preparation of high-quality domain datasets faces systemic challenges, and these issues directly constrain the final performance of large models, becoming a core bottleneck restricting the practical application of these models in various industries.

[0003] In current technological practices, data cleaning and preprocessing stages generally suffer from multiple technical deficiencies. Traditional manual processing methods are inefficient and lack standardized procedures. When faced with massive amounts of multi-source, heterogeneous data, operators often need to invest a significant amount of time in basic tasks such as format conversion and text splitting. Especially when processing unstructured documents such as PDFs and Markdowns, manually extracting key information is not only time-consuming and labor-intensive but also prone to compromising data consistency due to subjective judgment differences, thus affecting data quality. Furthermore, existing tools have weak adaptability to multimodal data, making it difficult to effectively integrate different modalities such as text, tables, and images, resulting in incomplete data feature extraction. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method for generating fine-tuning datasets for fine-tuning large multimodal models, to solve the technical problems of existing technologies that cannot handle different modal information and have low data quality. The method includes: Obtain cross-modal data from raw data in multiple file formats; Based on the cross-modal data, a three-dimensional domain knowledge graph containing subject classification, professional terminology, and entity relationships is constructed. Based on the aforementioned three-dimensional domain knowledge graph, question-answer pairs and thought chains are generated from the cross-modal data using an LLM (Large Language Model) federated architecture. Semantic alignment is performed on multiple modal data in the cross-modal data to obtain the fine-tuned dataset.

[0005] This invention also provides a fine-tuning dataset generation device for fine-tuning large multimodal models, to solve the technical problems of existing technologies that cannot handle different modal information and have low data quality. The device includes: The data acquisition module is used to obtain cross-modal data from raw data in multiple file formats; The knowledge graph construction module is used to construct a three-dimensional domain knowledge graph containing subject classification, professional terminology, and entity relationships based on the cross-modal data. The question-answer generation module is used to generate question-answer pairs and thought chains from the cross-modal data based on the three-dimensional domain knowledge graph and through the LLM model federated architecture. The data alignment module is used to perform semantic alignment on multiple morphological data in the cross-modal data to obtain the fine-tuned dataset.

[0006] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned methods for generating fine-tuning datasets for fine-tuning large multimodal models, thereby solving the technical problems of the prior art in being unable to handle different modal information and having low data quality.

[0007] This invention also provides a computer-readable storage medium storing a computer program that executes any of the above-described methods for generating fine-tuning datasets for fine-tuning large multimodal models, in order to solve the technical problems of the prior art being unable to handle different modal information and having low data quality.

[0008] Compared with existing technologies, the beneficial effects achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: It proposes an automated and intelligent method to obtain cross-modal data from raw data of various file formats, enabling intelligent processing of multi-source heterogeneous data in multiple formats, accurately extracting mixed content such as images, text, and tables, avoiding manual format conversion, text splitting, and other basic tasks, and preventing the impact of manual processing on data quality, thereby improving processing efficiency and data quality; and it proposes constructing a three-dimensional domain knowledge database based on cross-modal data, including subject classification, professional terminology, and entity relationships. The knowledge graph approach, tailored to domain characteristics, ensures the accuracy of knowledge representation. Based on a 3D domain knowledge graph, it generates question-answer pairs and thought chains from cross-modal data using an LLM model federated architecture. This reduces inference bias, improves the accuracy of solving complex problems, and enhances the precision of question-answer pairs. Finally, it performs semantic alignment on various modalities within the cross-modal data, achieving semantic matching between different modalities. This automatically associates heterogeneous, multimodal data such as text, images, and tables, making the fusion of multimodal information more natural and accurate, and providing high-quality data input for the training and inference of large multimodal models. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a method for generating fine-tuning datasets for fine-tuning large multimodal models, provided by an embodiment of the present invention; Figure 2 This is a flowchart illustrating a method for generating fine-tuning datasets for fine-tuning large multimodal models, as provided in an embodiment of the present invention. Figure 3 This is a structural block diagram of a computer device provided in an embodiment of the present invention; Figure 4 This is a structural block diagram of a fine-tuning dataset generation device for fine-tuning multimodal large models provided in an embodiment of the present invention. Detailed Implementation

[0011] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0012] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In this embodiment of the invention, a method for generating a fine-tuning dataset for fine-tuning a large multimodal model is provided, such as... Figure 1 As shown, the method includes: Step S101: Obtain cross-modal data from raw data in multiple file formats; Step S102: Based on the cross-modal data, construct a three-dimensional domain knowledge graph that includes subject classification, professional terminology, and entity relationships; Step S103: Based on the three-dimensional domain knowledge graph, generate question-answer pairs and thought chains from the cross-modal data using an LLM model federated architecture; Step S104: Perform semantic alignment on the multiple modal data in the cross-modal data to obtain the fine-tuned dataset (i.e., the fine-tuned dataset consists of cross-modal data, question-answer pairs, and thought chains).

[0014] Depend on Figure 1 As shown in the flowchart, this invention proposes an automated and intelligent method to obtain cross-modal data from raw data in various file formats. This enables intelligent processing of multi-source heterogeneous data in multiple formats, accurately extracting mixed content such as images, text, and tables, avoiding manual format conversion and text splitting, and preventing the impact of manual processing on data quality. This, in turn, improves processing efficiency and data quality. Furthermore, it proposes constructing a three-dimensional domain knowledge graph based on cross-modal data, including subject classifications, professional terms, and entity relationships. This is domain-specific and ensures the accuracy of knowledge representation. Based on the three-dimensional domain knowledge graph, a federated architecture using an LLM model is used to generate question-answer pairs and thought chains from the cross-modal data. This reduces inference bias, improves the accuracy of solving complex problems, and enhances the precision of question-answer pairs. Finally, semantic alignment is performed on multiple modalities within the cross-modal data to achieve semantic matching between different modalities. This automatically associates heterogeneous and multimodal data such as text, images, and tables, making the fusion of multimodal information more natural and accurate, and providing high-quality data input for the training and inference of large multimodal models.

[0015] In specific implementation, such as Figure 2 As shown, it can import multi-source heterogeneous data in various file formats such as PDF, Markdown, Excel, and Txt. Its adaptive parsing engine automatically recognizes these file formats and then uses deep learning-driven OCR technology to extract mixed content of images, text, and tables, constructing a representation matrix of cross-modal data containing text, tables, and images. This fully automated parsing method not only reduces manual intervention in data preprocessing but also significantly improves data usability and integrity, providing a solid foundation for subsequent data modeling and inference.

[0016] In practical implementation, to enhance the domain characteristics of data while ensuring the logical and semantic continuity of data organization, a method for constructing a three-dimensional domain knowledge graph based on cross-modal data is proposed. For example, based on the cross-modal data, a three-dimensional domain knowledge graph containing subject classifications, professional terms, and entity relationships is constructed, including: Based on the principle of context awareness, the text data in the cross-modal data is divided into blocks to obtain multiple text blocks; The knowledge density algorithm is used to identify the subject categories, professional terms, and entity relationships contained in each text block; Based on the subject classifications, professional terms, and entity relationships contained in all the text blocks, the three-dimensional domain knowledge graph is constructed.

[0017] In practice, to achieve accurate text segmentation and content structure optimization, ensure the logical and semantic continuity of data organization, and improve the quality and consistency of data cleaning, the text data in the cross-modal data is segmented based on the principle of context awareness, resulting in multiple text blocks (including chapter text blocks and topic text blocks), including: The title in the text data is identified based on a dynamic sliding window algorithm, and the content of different titles is divided into different chapter text blocks. For each chapter text block, determine the topic of each paragraph, and divide paragraphs with the same topic into text blocks with the same topic.

[0018] For example, intelligent text segmentation can be achieved based on algorithms using dynamic sliding windows (where the length of the sliding window can be dynamically adjusted according to the size of the segmented text blocks or the amount of text, allowing text segmentation to support custom minimum and maximum text fragment lengths). Figure 2 As shown, Markdown headings (#, ##) are identified as the basis for dividing text blocks into chapters. Then, large models (such as GPT-4 and Claude) are used to determine the thematic consistency of each paragraph, further optimizing the segmentation points and dividing paragraphs with the same theme into text blocks of the same theme, thus achieving outline extraction. By combining dynamic sliding window algorithms and large model inference techniques, accurate text segmentation and content structure optimization are achieved. By automatically identifying headings and chapter division points, and combining them with thematic consistency analysis, the information fragmentation problem that may be caused by traditional block-based methods can be effectively avoided. This optimization strategy based on intelligent block-based segmentation ensures the logical and semantic continuity of data organization, improving the quality and consistency of data cleaning.

[0019] In practical implementation, knowledge density calculation methods (such as calculating the frequency of each word appearing in the text for subject classifications, professional terms, entities, etc., and then selecting words with frequencies greater than a threshold to construct a three-dimensional domain knowledge graph) can be used to automatically identify core concepts (such as subject classifications, professional terms, entities, etc.) in the text. Figure 2As shown, LLM-driven domain tree generation technology is then used to construct a three-dimensional domain knowledge graph (in the form of a domain tree) that includes subject classification, professional terminology, and entity relationships, supporting user-defined node weight adjustments. Through domain tree generation technology and knowledge density calculation methods, multi-dimensional domain knowledge graphs can be automatically identified and constructed. This method not only automatically classifies subject areas and extracts professional terminology but also supports user-defined node weights, thus ensuring the accuracy and flexibility of knowledge representation. Combined with large-scale model reasoning capabilities, this knowledge modeling approach is widely applicable to various complex data processing scenarios, providing high-quality semantic support for intelligent question answering, reasoning, and decision-making.

[0020] In practice, users often face difficulties in constructing a domain labeling system when processing segmented domain data—they need to consider both the deep integration of industry knowledge graphs and the reasonableness of label granularity. To further improve the quality of the dataset, a method for optimizing question-answer pairs is proposed. For example, based on the aforementioned three-dimensional domain knowledge graph, question-answer pairs are generated from the cross-modal data using an LLM model federated architecture, including: For each text block, the three-dimensional domain knowledge graph is input into multiple base models in the LLM model federated architecture, and each base model outputs its own question-answer pair, resulting in multiple question-answer pairs; The optimal question-answer pair is selected from multiple question-answer pairs through a voting mechanism and used as the question-answer pair for this text block.

[0021] In practical implementation, an LLM model federated architecture can be built, such as integrating six types of base models including Ollama and GPT-4. Each base model outputs its own question-answer pairs based on a three-dimensional domain knowledge graph. Then, a voting mechanism is used to select the optimal question-answer pair from the outputs of the six base models (e.g., Figure 2 As shown, the optimal answers are selected for questions 1, 2, and 3 to generate question-answer pairs, which serve as the question-answer pairs for this text block. This achieves the determination of question-answer pairs for text blocks through multi-model collaborative reasoning. Compared with traditional single-model reasoning methods, this application can effectively reduce reasoning bias and improve the accuracy of solving complex problems. Figure 2 As shown, based on the generated question-and-answer pairs, different windows can be provided to users to view the problem, such as list view, domain view, question modification, question deletion, etc., with each window corresponding to a different perspective.

[0022] In practice, question-and-answer pairs can be generated for each chapter text block and topic text block to achieve hierarchical structure of question-and-answer pairs, thereby improving the accuracy of the dataset.

[0023] In practice, when constructing the thought chain (COT) dataset required for the inference model, existing technologies have not yet formed a standardized generation paradigm. Developers often need to repeatedly experiment with different prompting engineering strategies, and this trial-and-error process significantly increases time costs. For example... Figure 2 As shown, this application proposes a recursive chain of reasoning (COT) generation algorithm to automate the construction of complex reasoning paths. The COT reasoning process is adaptively adjustable and supports progressive reasoning strategies to enhance the logicality of the answer. For example, based on the inherent structure and logical relationships of the problem, the problem is broken down into multiple logically related sub-problems (i.e., logical relationships exist between the sub-problems), and a dependency tree between the problem and its sub-problems is constructed. Following the hierarchy in the dependency tree, reasoning begins from the lowest-level sub-problem to determine the sub-answer of the lowest-level sub-problem. This sub-answer is then used as the input for reasoning the next lowest-level sub-problem to determine its answer, and so on. Different lowest-level sub-problems form different paths to the problem. The reasoning processes of the sub-problems on different paths are combined to form the reasoning process of that path, and the reasoning processes of different paths form the COT.

[0024] In specific implementation, an optimization method for question-answer pairs is also proposed. For example, for all the question-answer pairs of the text blocks, each question-answer pair is converted into a machine-readable numerical form to obtain the TF-IDF vector of each question-answer pair (TF is the single-file term frequency, IDF is the inverse text frequency, and TF-IDF is represented as TF * IDF). The similarity between the TF-IDF vectors of every two question-answer pairs is calculated. If the similarity reaches a preset threshold, the two question-answer pairs are determined to be duplicates and are merged. This implements a hybrid duplicate detection mechanism, combining TF-IDF vector space modeling and semantic similarity calculation, which can achieve a deduplication rate of >95% for cross-batch generated question-answer pairs.

[0025] In practice, existing cleaning methods often employ simple content stripping strategies when processing documents with mixed text and images, leading to the loss of text-image association information. For example, in the processing of technical manuals, separating and parsing diagrams and explanatory text can damage the integrity of knowledge representation. Furthermore, the ability to mine cross-document knowledge associations is insufficient, making it difficult to construct a logically hierarchical knowledge system, which directly affects the reasoning ability of the fine-tuned model. To further improve the quality of the dataset, this application proposes semantic alignment of multiple modalities in the cross-modal data, including: The text data in the cross-modal data is processed to generate context-dependent embedding vectors; visual features are extracted from the image data in the cross-modal data and converted into vector representations; the tabular data in the cross-modal data is processed to generate structured representations. The embedding vector and the vector representation are input into a text-image attention model, which outputs a semantic matching result between text and image; the embedding vector and the structured representation are input into a text-table attention model, which outputs a semantic matching result between text and rows and columns in a table.

[0026] In practical implementation, to achieve cross-modal data alignment, firstly, text data in the cross-modal data is processed using Transformers (such as BERT, T5) to generate context-sensitive embedding vectors; image data (image encoding) in the cross-modal data is extracted using CNN or Vision Transformer (ViT) and converted into vector representations; tabular data (table encoding) in the cross-modal data is processed using TabNet or TAPAS (Transformer for Tables, table parser) to generate structured representations (such as representations including rows and columns); secondly, a cross-modal attention mechanism is constructed: 1) Constructing a text-image attention model, such as using CLIP (Contrastive Language-Image Pretraining, a multimodal pre-training model) as the text-image attention model, inputting the embedding vectors and vector representations into the text-image attention model for text-image matching. The semantic matching results of text and image include making the text focus on (or match) key regions in the image, and determining which parts of the two are most relevant, etc.; 2) Constructing a text-table attention model, such as using LlamaIndex + TAPAS, as a text-table attention model, takes embedded vectors and structured representations as input, parses the table information, and outputs semantic matching results between the text and the rows and columns of the table, such as alignment of text semantics with table rows and columns. It achieves semantic matching between data of different modalities, automatically associating heterogeneous data such as text, images, and tables, making the fusion of multimodal information more natural and accurate, and providing high-quality data input for the training and inference of large multimodal models.

[0027] In practical implementation, to facilitate efficient and accurate data updates, methods for updating datasets are proposed, for example, Set version information for the fine-tuned dataset; Determine the difference between the newly added data and the current version of the fine-tuned dataset, wherein the difference is data in the newly added data that is not included in the current version of the fine-tuned dataset; The difference data is updated to the current version of the fine-tuning dataset using semantic fingerprinting technology. Specifically, this involves the following steps: 1. Extract the semantic fingerprint of each data unit from the current version of the fine-tuning dataset, where the data unit is a sample or sample cluster in the dataset; 2. Based on a similarity calculation algorithm, the semantic fingerprint of the difference data is compared with the semantic fingerprint of each data unit, and the data unit with the highest similarity is determined as the update target; 3. When the similarity corresponding to the updated target is lower than a preset threshold, the difference data is added to the fine-tuning dataset; 4. When the similarity corresponding to the update target is not lower than the preset threshold, the difference data is merged into the data unit corresponding to the update target to generate a new version of the fine-tuned dataset, and the new version replaces the current version.

[0028] This method achieves precise location of differential data and target data units through semantic fingerprint similarity matching, avoiding full replacement, reducing data redundancy, and improving the accuracy and efficiency of fine-tuning dataset updates.

[0029] Specifically, version control modules and difference analysis engines can be developed to support iterative updates of datasets and historical version backtracking. Semantic fingerprint technology can be used to achieve automatic fusion of incremental data, ensuring the traceability and stability of data cleaning.

[0030] In practical implementation, existing technical solutions also have significant shortcomings in quality assessment. Most systems only provide simple statistical indicators (such as the number of question-answer pairs, character length, etc.), lacking automated assessment of deeper features such as semantic quality and logical consistency. To further improve the quality of the dataset, this application can perform quality assessment and optimization, deploying a multi-dimensional quality assessment matrix, including nine assessment indicators such as logical consistency scoring, factual accuracy verification, and language fluency detection. Figure 2 As shown, the dataset is scored using evaluation metrics to improve its quality, allowing for data correction based on the evaluation results and enhancing the reliability of the data analysis.

[0031] In practice, existing manual scoring processes are often disconnected from data processing, hindering real-time feedback and dynamic optimization. This lack of quality monitoring directly leads to low-quality data being mixed into the training set, resulting in model performance degradation. For example... Figure 2 As shown, this application can also enhance the dataset through human intervention, for example, by designing a dual-channel feedback system to support online expert annotation and quality scoring (i.e., human scoring), developing a data cleaning model based on active learning, and transforming human feedback into parameter optimization of automated cleaning rules.

[0032] In practical implementation, regarding model compatibility, existing systems typically only support a limited number of large language model interfaces, making it difficult to adapt to the rapidly iterating technological ecosystem. Users face complex adaptation challenges when integrating custom models. To meet the need for the dataset to be used for fine-tuning different large models, the dataset export needs to satisfy various format conversions. This application's dataset can undergo multi-format adaptive conversion, such as constructing a format conversion template library containing eight frameworks including Alpaca and ShareGPT, and using Abstract Syntax Tree (AST) parsing technology to achieve lossless data structure conversion.

[0033] In practical implementation, the above-mentioned method for generating fine-tuning datasets for fine-tuning multimodal large models can achieve full-process visual monitoring, such as building a data processing graph visualization system to display the quality indicators and processing progress of each stage in real time, providing anomaly detection and early warning and root cause analysis functions, forming a closed-loop quality control system.

[0034] In practical implementation, the aforementioned method for generating fine-tuning datasets for multimodal large-scale model fine-tuning is an adaptive and automated cleaning method for multimodal large-scale data. It aims to solve core problems encountered in fine-tuning large models in fields such as aviation, healthcare, finance, and education, including low data quality, insufficient processing efficiency, and difficulties in multimodal collaboration. To overcome the shortcomings of existing technologies, such as fragmented processing, semantic separation, and excessive reliance on manual intervention, the method of this invention constructs a closed-loop system encompassing "parsing-blocking-modeling-generation-evaluation" to achieve intelligent transformation from raw data to high-quality fine-tuning datasets.

[0035] In practical implementation, taking aerospace data cleaning as an example, the above-mentioned method for generating fine-tuning datasets for multimodal large model fine-tuning may include the following steps: This invention provides an adaptive multi-source data adaptive automated cleaning method, which, in conjunction with a specific case, includes the following steps: S1. Importing and parsing multi-source heterogeneous data: Taking an airline's desire to optimize flight safety data management as an example, its data sources include flight logs (PDF format), maintenance reports (Excel format), pilot feedback (Markdown format), and accident investigation documents (scanned images). This system utilizes an adaptive parsing engine to automatically identify and parse these multi-source data, and extracts text information from the scanned images using OCR technology. It then constructs a cross-modal data representation matrix containing text, tables, and images, providing standardized input for subsequent data processing.

[0036] S2. Context-aware intelligent text chunking: Since flight logs and accident reports typically contain a large amount of unstructured text, this system employs a dynamic sliding window algorithm for text segmentation, combined with Markdown headings (#, ##) for chapter division, resulting in chapter text blocks, topic text blocks, etc. Taking a flight accident report as an example, the system automatically identifies key sections such as "Pre-flight checks," "In-flight anomalies," and "Emergency handling," divides them into chapter text blocks, and uses a large model (such as GPT-4) to analyze the thematic consistency of each paragraph, grouping paragraphs with the same theme into the same topic text block, ensuring that the segmentation points are logically consistent and improving information retrieval and utilization efficiency.

[0037] S3. Multi-level knowledge structure modeling: To better analyze accident causes and flight safety data, the system employs LLM-driven domain tree generation technology to automatically identify core concepts such as "engine failure," "weather factors," and "pilot error," and constructs a three-dimensional domain knowledge graph that includes subject classifications, professional terminology, and entity relationships. For example, in a specific in-flight fire case, the system automatically categorizes key factors such as "fuel leak" and "fire detection system response time," and establishes correlations to facilitate subsequent analysis and tracing.

[0038] S4. Multi-model collaborative reasoning construction: During safety analysis, different experts may have different judgments about the cause of an accident. The system's LLM model federated architecture integrates multiple base models such as Ollama and GPT-4, and uses a voting mechanism to select the optimal question-answer pair. For example, in the analysis of an incident in which landing gear failed to deploy properly, one model identified it as a "hydraulic system failure," while another model identified it as a "sensor false alarm." Through multi-model voting and a thought chain (COT) recursive generation algorithm, the system ultimately determines the most likely cause of the accident and provides a detailed reasoning process.

[0039] S5. Question-answer pair optimization and data deduplication: When processing multiple batches of flight data, duplicate question-and-answer content can negatively impact data quality. This system employs a hybrid deduplication mechanism, combining TF-IDF vector space modeling with semantic similarity calculation to ensure deduplication of similar content. For example, in pilot feedback from multiple flights, the issue of "severe weather causing turbulence" appears on multiple flights; the system automatically merges similar feedback, ensuring efficient management and optimization of the question-and-answer database.

[0040] S6. Dynamic Tag Management System: Because aviation safety data involves numerous terms and classifications, manual annotation can lead to label conflicts or redundancy. The system develops a label conflict detection algorithm and domain adapter to support user-defined labeling systems and integrates with LLM for automatic annotation. For example, for issues related to "fuel systems," the system can automatically identify and standardize labels such as "fuel leak" and "abnormal fuel pressure," avoiding duplicate or incorrect labeling and improving data consistency and retrieval efficiency.

[0041] S7. Cross-modal data alignment: Flight data typically includes multimodal information such as text descriptions, instrument readings, and image recordings. The system employs a cross-modal attention mechanism to match text, images, and tables. For example, in the analysis of a lightning strike incident, the system automatically matches the pilot's description of "the lightning strike occurred on the left wing" with the lightning strike photograph taken by the airborne camera, and combines this with table data to analyze electrical system readings before and after the lightning strike, ensuring data alignment and improving analysis accuracy.

[0042] S8. Quality Assessment and Optimization: The system deploys a multi-dimensional quality assessment matrix, including evaluation indicators such as logical consistency scoring, factual accuracy verification, and language fluency detection. For example, in an automatically generated flight data report, if the system detects a discrepancy between a conclusion and the data recorded by the black box, it automatically marks the error and provides correction suggestions to improve the reliability of data analysis.

[0043] S9. Incremental data update: Aviation safety data requires continuous updates and optimization. This system supports version control and historical review of datasets, and uses semantic fingerprinting technology to automatically fuse incremental data. For example, when an airline updates its 2023 flight safety report, the system automatically detects differences from the 2022 data and only updates the newly added and differing data, avoiding duplicate storage and improving data management efficiency.

[0044] S10. Multi-format adaptive conversion and visualization: To meet diverse research and regulatory needs, the system has built a library of data format conversion templates, such as Alpaca and ShareGPT, and employs AST parsing technology to achieve lossless data conversion. For example, when submitting flight safety data to regulatory agencies, the system can automatically convert it to JSON or Excel format, while providing visual graphs to intuitively display safety trends and key risk factors, thereby improving data utilization and decision support capabilities.

[0045] The fine-tuning dataset generation method for multimodal large model fine-tuning proposed in this application combines intelligent parsing, knowledge modeling, multimodal alignment, dynamic optimization and other technologies, which significantly improves the cleaning quality and analysis efficiency of multi-source heterogeneous data. It is widely applicable to complex data scenarios such as aviation safety, medical care, and finance, and provides an efficient and reliable solution for intelligent decision-making and automated data management.

[0046] In this embodiment, a computer device is provided, such as... Figure 3 As shown, it includes a memory 301, a processor 302, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned methods for generating fine-tuning datasets for fine-tuning multimodal large models.

[0047] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0048] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that performs any of the above-described methods for generating fine-tuning datasets for fine-tuning large multimodal models.

[0049] Specifically, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0050] Based on the same inventive concept, this invention also provides a fine-tuning dataset generation apparatus for fine-tuning multimodal large models, as described in the following embodiments. Since the principle of the fine-tuning dataset generation apparatus for fine-tuning multimodal large models is similar to that of the fine-tuning dataset generation method for fine-tuning multimodal large models, the implementation of the fine-tuning dataset generation apparatus for fine-tuning multimodal large models can refer to the implementation of the fine-tuning dataset generation method for fine-tuning multimodal large models, and will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0051] Figure 4 This is a structural block diagram of a fine-tuning dataset generation device for fine-tuning large multimodal models according to an embodiment of the present invention, such as... Figure 4 As shown, it includes: The data acquisition module 401 is used to obtain cross-modal data from raw data in multiple file formats; The graph construction module 402 is used to construct a three-dimensional domain knowledge graph containing subject classification, professional terminology, and entity relationships based on the cross-modal data. Question-answer generation module 403 is used to generate question-answer pairs and thought chains from the cross-modal data based on the three-dimensional domain knowledge graph and through the LLM model federated architecture; The data alignment module 404 is used to perform semantic alignment on multiple morphological data in the cross-modal data to obtain the fine-tuned dataset.

[0052] The embodiments of this invention achieve the following technical effects: First, it proposes an automated and intelligent method to obtain cross-modal data from raw data in multiple file formats. This enables intelligent processing of multi-source heterogeneous data in various formats, accurately extracting mixed content such as images, text, and tables, avoiding manual format conversion and text splitting, and preventing the impact of manual processing on data quality. This improves processing efficiency and data quality. Second, it proposes constructing a three-dimensional domain knowledge graph based on cross-modal data, including subject classifications, professional terms, and entity relationships. This is domain-specific, ensuring the accuracy of knowledge representation. Based on the three-dimensional domain knowledge graph, a federated architecture using an LLM model generates question-answer pairs and thought chains from the cross-modal data, reducing inference bias, improving the accuracy of solving complex problems, and enhancing the precision of question-answer pairs. Finally, it performs semantic alignment on multiple modalities within the cross-modal data, achieving semantic matching between different modalities. This automatically associates heterogeneous and multimodal data such as text, images, and tables, making the fusion of multimodal information more natural and accurate, providing high-quality data input for the training and inference of large multimodal models.

[0053] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular combination of hardware and software.

[0054] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fine-tuning dataset generation method for multi-modal large model fine-tuning, characterized in that, include: Obtain cross-modal data from raw data in multiple file formats; Based on the cross-modal data, a three-dimensional domain knowledge graph containing subject classification, professional terminology, and entity relationships is constructed. Based on the three-dimensional domain knowledge graph, question-answer pairs and thought chains are generated from the cross-modal data using an LLM model federated architecture; Semantic alignment is performed on multiple modal data in the cross-modal data to obtain the fine-tuned dataset.

2. The fine-tuning dataset generation method for multi-modal large model fine-tuning according to claim 1, wherein, Based on the aforementioned cross-modal data, a three-dimensional domain knowledge graph is constructed, including subject classification, professional terminology, and entity relationships, comprising: Based on the principle of context awareness, the text data in the cross-modal data is divided into blocks to obtain multiple text blocks; The knowledge density algorithm is used to identify the subject categories, professional terms, and entity relationships contained in each text block; Based on the subject classifications, professional terms, and entity relationships contained in all the text blocks, the three-dimensional domain knowledge graph is constructed.

3. The method for generating fine-tuning datasets for fine-tuning large multimodal models as described in claim 2, characterized in that, Based on the principle of context awareness, the text data in the cross-modal data is divided into blocks, resulting in multiple text blocks, including: The title in the text data is identified based on a dynamic sliding window algorithm, and the content of different titles is divided into different chapter text blocks. For each chapter text block, determine the topic of each paragraph, and divide paragraphs with the same topic into text blocks with the same topic.

4. The method for generating fine-tuning datasets for fine-tuning large multimodal models as described in claim 2, characterized in that, Based on the aforementioned 3D domain knowledge graph, question-answer pairs are generated from the cross-modal data using an LLM model federated architecture, including: For each text block, the three-dimensional domain knowledge graph is input into multiple base models in the LLM model federated architecture, and each base model outputs its own question-answer pair, resulting in multiple question-answer pairs; The optimal question-answer pair is selected from multiple question-answer pairs through a voting mechanism and used as the question-answer pair for this text block.

5. The method for generating fine-tuning datasets for fine-tuning large multimodal models as described in claim 4, characterized in that, Also includes: For all the question-answer pairs in the text blocks, each question-answer pair is converted into a machine-readable numerical form to obtain the TF-IDF vector of each question-answer pair; Calculate the similarity between the TF-IDF vectors of every two question-answer pairs. If the similarity reaches a preset threshold, then the two question-answer pairs are determined to be duplicates and are merged.

6. The method for generating a fine-tuning dataset for fine-tuning a multimodal large model as described in any one of claims 1 to 5, characterized in that, Semantic alignment of multiple modalities in the cross-modal data includes: The text data in the cross-modal data is processed to generate context-dependent embedding vectors; visual features are extracted from the image data in the cross-modal data and converted into vector representations; the tabular data in the cross-modal data is processed to generate structured representations. The embedding vector and the vector representation are input into a text-image attention model, which outputs a semantic matching result between text and image; the embedding vector and the structured representation are input into a text-table attention model, which outputs a semantic matching result between text and rows and columns in a table.

7. The method for generating a fine-tuning dataset for fine-tuning a multimodal large model as described in any one of claims 1 to 5, characterized in that, Also includes: Set version information for the fine-tuned dataset; Determine the difference between the newly added data and the current version of the fine-tuned dataset, wherein the difference is data in the newly added data that is not included in the current version of the fine-tuned dataset; The difference data is updated to the current version of the fine-tuning dataset using semantic fingerprinting technology to generate a new version of the fine-tuning dataset, and the new version is used as the current version.

8. A device for generating fine-tuning datasets for fine-tuning large multimodal models, characterized in that, include: The data acquisition module is used to obtain cross-modal data from raw data in multiple file formats; The knowledge graph construction module is used to construct a three-dimensional domain knowledge graph containing subject classification, professional terminology, and entity relationships based on the cross-modal data. The question-answer generation module is used to generate question-answer pairs and thought chains from the cross-modal data based on the three-dimensional domain knowledge graph and through the LLM model federated architecture. The data alignment module is used to perform semantic alignment on multiple morphological data in the cross-modal data to obtain the fine-tuned dataset.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the fine-tuning dataset generation method for fine-tuning a multimodal large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the fine-tuning dataset generation method for fine-tuning a multimodal large model as described in any one of claims 1 to 7.

Images