A method for building a multimodal large model based on low code

Abstract

The present invention relates to a method for constructing a multimodal large model based on low code, and belongs to the field of artificial intelligence. It comprises the following steps: S1: acquiring multimodal data, wherein the multimodal data comprises structured data and unstructured data; S2: constructing a structured data processing module, wherein the structured data passes through the structured data processing module to obtain aligned structured data; S3: constructing an unstructured data processing module, wherein the unstructured data passes through the unstructured data processing module to obtain aligned unstructured data; S4: applying the aligned structured data and the aligned unstructured data to a base large model. The present invention realizes alignment of structured data by integrating PySpark table merging technology, AutoML and NL2SQL into the structured data processing module, and integrates Swin Transformer and CLIP models into the unstructured data processing module, thereby realizing alignment of unstructured data and improving the performance of the multimodal large model in multimodal tasks.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to a method for constructing a multimodal large model based on low code. Background Art

[0002] The current mainstream low-code large model platforms mainly rely on open source communities and open source platforms. These open source platforms and projects provide developers with flexible tools and frameworks, enabling them to more easily leverage the capabilities of large language models and build various applications. Although these platforms have their own focuses, they have jointly promoted the application and popularization of large models in low-code environments.

[0003] Hugging Face provides a powerful open source library for handling natural language processing (NLP) tasks. Although it is not a low-code platform itself, Hugging Face enables developers to easily integrate and use large language models through simple APIs and a rich library of pre-trained models. Combined with some low-code tools, the development and deployment of NLP models can be greatly simplified. Lobe is an open source project supported by Microsoft that aims to help users train machine learning models through a simple drag-and-drop interface. Although it currently mainly supports image classification, its low-code development concept and interface design provide a foundation for applications that may be expanded to large models in the future. Haystack is an open source framework for building question-answering systems that supports the use of large language models (such as BERT, RoBERTa, etc.) to process natural language queries. It provides simplified interfaces and tools that enable developers to quickly build and deploy question-answering systems based on large models.

[0004] Although traditional large multimodal models have achieved great success in practice and usually perform well in processing unstructured data (such as images, text, and speech), their performance on structured data (such as tabular data) is not as good as models specifically designed to process structured data. This limits their effectiveness in application scenarios that require processing both structured and unstructured data. Structured data and unstructured data often require different feature representation methods, and multimodal models need to be able to effectively integrate feature representations of different data types to improve their performance in multimodal tasks. Summary of the invention

[0005] The purpose of the present invention is to provide a low-code based multimodal large model construction method, which improves the processing performance of the multimodal large model for structured data and unstructured data.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] In the first aspect, a method for building a multimodal large model based on low code includes the following steps:

[0008] S1: Acquire multimodal data, where the multimodal data includes structured data and unstructured data;

[0009] S2: constructing a structured data processing module, the structured data processing module includes a table merging module, a model flow building module and a structured data alignment module, the structured data is subjected to a table merging operation by the table merging module to obtain a feature wide table, the feature wide table is subjected to the model flow building module to obtain a decision result of the feature wide table, the decision result is input together with the feature attributes in the feature wide table into the structured data alignment module to obtain aligned structured data;

[0010] S3: constructing an unstructured data processing module, the unstructured data processing module comprising a target detection module and an unstructured data alignment module, the unstructured data undergoes feature extraction by the target detection module to obtain feature representation of the unstructured data, and the feature representation is input into the unstructured data alignment module to obtain aligned unstructured data;

[0011] S4: Apply the aligned structured data and the aligned unstructured data to the base large model to complete the construction of the multimodal large model.

[0012] Furthermore, in step S1, the structured data is a data table, and the unstructured data is a picture.

[0013] Furthermore, the table merging module in step S2 adopts PySpark table merging technology, takes PySpark as a visualization operator of the low-code platform, performs SQL-based visualization editing on the operator, completes the table merging operation, and obtains a feature wide table, which specifically includes:

[0014] PySpark visualization: PySpark operator visualization technology is used to pre-process the structured data. PySpark operators provide a unified DataFrame API interface to implement table merging operations. Multiple tables are combined into a new structured data table, i.e., a feature wide table, through visualization SQL and PySpark.

[0015] In the process implementation of the PySpark operator, a union operation is used to merge tables. The union operation is used to merge two or more data sets into a new data set. The data set is in the form of RDD or DataFrame.

[0016] Furthermore, the model process construction module in step S2 specifically includes: constructing an AutoML process by means of automated feature engineering, wherein the automated feature engineering includes an automated feature combination operator, and the automated feature combination operator is used to perform feature combination analysis on the feature wide table, generate features based on the combination method, and evaluate the importance of the generated features, and finally obtain a decision result of the feature wide table.

[0017] Furthermore, the combination-based method is a combine function, and the automatic feature combination operator uses a bucketing technique.

[0018] Furthermore, the structured data alignment module in step S2 uses NL2SQL technology to convert the natural language in the structured data field into an executable structured query language in a relational database to obtain aligned structured data.

[0019] Furthermore, the target detection module in step S3 uses a pre-trained Swin Transformer to perform feature extraction on the unstructured data to obtain the feature representation, and the unstructured data alignment module uses a CLIP model to align text features and image features in the feature representation to obtain aligned unstructured data.

[0020] Furthermore, the pre-trained Swin Transformer is pre-trained using the public COCO dataset.

[0021] In a second aspect, a device for building a multimodal large model based on low code includes:

[0022] Data acquisition unit: used to acquire multimodal data;

[0023] The first construction unit is used to construct a structured data processing module, which includes a table merging module, a model flow construction module and a structured data alignment module. The structured data is subjected to a table merging operation by the table merging module to obtain a feature wide table. The feature wide table is subjected to the model flow construction module to obtain a decision result of the feature wide table. The decision result is input into the structured data alignment module together with the feature attributes in the feature wide table to obtain aligned structured data.

[0024] A second construction unit is used to construct an unstructured data processing module, wherein the unstructured data processing module includes a target detection module and an unstructured data alignment module. The unstructured data is subjected to feature extraction by the target detection module to obtain feature representation of the unstructured data, and the feature representation is input into the unstructured data alignment module to obtain aligned unstructured data.

[0025] The third construction unit is used to apply the aligned structured data and the aligned unstructured data to the base large model to complete the construction of the multimodal large model.

[0026] In a third aspect, an electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the low-code-based multimodal large model building method described in the first aspect are implemented.

[0027] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the low-code-based multimodal large model building method described in the first aspect.

[0028] The effects of the present invention are as follows: by constructing a structured data processing module and an unstructured data processing module, PySpark table merging technology, AutoML and NL2SQL are integrated into the structured data processing module to achieve structured data alignment; Swin Transformer and CLIP models are integrated into the unstructured data processing module to achieve unstructured data alignment, thereby improving the performance of multimodal large models in multimodal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0030] Figure 1 It is a data training flow chart of the present invention;

[0031] Figure 2 This is a table merging operation based on PySpark of the present invention;

[0032] Figure 3 It is a graphical interface used in fine-tuning the method of the present invention;

[0033] Figure 4 The training results of the structured data of the present invention in the multimodal large model are presented;

[0034] Figure 5 The results of the model of the present invention analyzing the trees affected by insects are shown;

[0035] Figure 6 This is a picture of a tree infested with insects with red dots in the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Example 1

[0038] The disclosed embodiment provides a method for building a multimodal large model based on low code.

[0039] like Figure 1 As shown in FIG. 1 , it is a data processing flow chart of the present invention. This embodiment includes the following steps:

[0040] S1: Acquire multimodal data, where the multimodal data includes structured data and unstructured data;

[0041] In this embodiment, the structured data is a data table, and the unstructured data is a picture.

[0042] S2: constructing a structured data processing module, the structured data processing module includes a table merging module, a model flow building module and a structured data alignment module, the structured data is subjected to a table merging operation by the table merging module to obtain a feature wide table, the feature wide table is subjected to the model flow building module to obtain a decision result of the feature wide table, the decision result is input together with the feature attributes in the feature wide table into the structured data alignment module to obtain aligned structured data;

[0043] In this embodiment, the table merging module adopts PySpark table merging technology, uses PySpark as a visualization operator of the low-code platform, performs SQL-based visualization editing on the operator, completes the table merging operation, and obtains a feature wide table, specifically including: PySpark visualization, using PySpark operator visualization technology to pre-process the structured data, and the PySpark operator provides a unified DataFrame API interface, such as Figure 2 As shown, multiple tables are combined into a new structured data table, i.e., a feature wide table, through visual SQL and PySpark, and the PySpark operator can be customized according to the needs to write a visualization process;

[0044] In the process implementation of the PySpark operator, we use the union operation to merge tables. The union operation is used to merge two or more data sets (usually RDD or DataFrame) into a new data set. In order to improve the efficiency and flexibility of the union operation, it can be designed as an operator so that it can better adapt to complex distributed data processing tasks. The following is a detailed design description of the union operator.

[0045] The core idea of ​​operator design is to abstract the union operation into an extensible high-order operator, so that it can not only perform basic merge operations, but also support more advanced functions through operator configuration and combination, such as data distribution optimization, dynamic load balancing, and customized merge logic.

[0046] Operator interface definition: The operatorized union operator needs to provide a unified interface for users to call and configure its behavior. The union operator interface can be designed as follows:

[0047] class UnionOperator:

[0048] def __init__(self, *dataframes, repartition=1, balance_load=1,custom_merge_fn=None):

[0049] self.dataframes = dataframes

[0050] self.repartition = repartition

[0051] self.balance_load = balance_load

[0052] self.custom_merge_fn = custom_merge_fn

[0053] def execute(self):

[0054] result_df = self._preprocess_and_union()

[0055] if self.custom_merge_fn:

[0056] result_df = self.custom_merge_fn(result_df)

[0057] return result_df

[0058] def _preprocess_and_union(self):

[0059] if self.repartition:

[0060] self.dataframes = [df.repartition("id") for df in self.dataframes]

[0061] result_df = self.dataframes[0]

[0062] for df in self.dataframes[1:]:

[0063] result_df = result_df.union(df)

[0064] if self.balance_load:

[0065] result_df = self._balance_load(result_df)

[0066] return result_df

[0067] def _balance_load(self, df):

[0068] return df

[0069] Operator implementation details: Automatic repartitioning. In the _preprocess_and_union() method, the repartition parameter determines whether to repartition each input DataFrame. The purpose of repartitioning is to ensure that the number of partitions of each data set is consistent before merging, so as to avoid data skew. The basis for repartitioning can be a specific column (such as the id column) or a partitioning strategy dynamically determined according to the amount of data; basic merge operation. The basic union operation is performed after preprocessing (such as repartitioning). The operatorized design enables the union operation to flexibly merge multiple DataFrames in sequence; load balancing optimization is achieved through the _balance_load() method. In an actual distributed environment, some partitions may become too large or too small after the union operation. To avoid this imbalance, the _balance_load() method can dynamically adjust the load of the partitions; custom merge logic. In order to meet specific business needs, the operatorized design supports users to pass in custom merge functions (custom_merge_fn). For example, users can customize the aggregation of certain columns during the merge, or deduplicate the merged data.

[0070] Operator execution and results: When using the UnionOperator operator, users only need to configure parameters and call the execute() method to complete complex data merging operations. This process includes data preprocessing, merging, and subsequent optimization steps such as load balancing.

[0071] union_operator = UnionOperator(df1, df2, repartition=1, balance_load=1)

[0072] result_df = union_operator.execute()

[0073] The result_df returned in the end is an optimized and processed DataFrame that contains the data of all input datasets and has been balanced to ensure efficient execution in a distributed environment.

[0074] Extension and application: The operatorized union operation is not limited to simple DataFrame merging, but can also be combined with other operators to form a more complex data processing flow. For example, after the union operation, you can continue to perform groupBy, filter and other operations to form a complete data processing pipeline.

[0075] In addition, the operator-based design can be easily extended to support more optimization strategies or special processing logic, for example, further refine the load balancing mechanism or add support for streaming data merging.

[0076] In this embodiment, the model process construction module specifically includes: building an AutoML process by means of automated feature engineering, wherein the automated feature engineering includes an automated feature combination operator, wherein the automated feature combination operator is used to perform feature combination analysis on the feature wide table, generate features based on a combination method (i.e., a combine function), and evaluate the importance of the generated features, and finally obtain a decision result of the feature wide table; the automatic feature combination mainly includes the following features: the operator uses a bucketing technique, and the output results are all discretized nonlinear features; the output includes both first-order original features and other-order combined features (e.g., 1, 2...), up to 5 orders; slotwise_auc is used as a scoring indicator for the feature, and a higher score indicates a higher feature importance.

[0077] The results of feature extraction can be returned to the corresponding feature extraction operator script configuration box and pasted for application. Currently, only copying of all features before the corresponding feature is supported.

[0078] The logic of the automated feature engineering algorithm is as follows:

[0079] Input: Initial feature set I, the number of combined features k selected in each iteration

[0080] output: candidate feature set C

[0081] 1 Initialization: C is the pairwise combination of all features in I, and the feature set S = I has been selected;

[0082] 2 while does not meet the termination condition do

[0083] 3. Evaluate the importance of each feature in C;

[0084] 4 Select k most important features from C and record the result as set T;

[0085] 5 Remove the features in T from the set C;

[0086] 6. Combine the features in T and S in pairs, and record the result as set O;

[0087] 7 Put the features in O into C;

[0088] 8 Put the features in T into S;

[0089] 9 Clear T, O;

[0090] 10 end.

[0091] In this embodiment, the structured data alignment module uses NL2SQL technology to convert the natural language in the structured data field into an executable structured query language in a relational database to obtain aligned structured data.

[0092] S3: constructing an unstructured data processing module, the unstructured data processing module comprising a target detection module and an unstructured data alignment module, the unstructured data undergoes feature extraction by the target detection module to obtain feature representation of the unstructured data, and the feature representation is input into the unstructured data alignment module to obtain aligned unstructured data;

[0093] In this embodiment, a pre-trained Swin Transformer is used to extract features from the unstructured data to obtain the feature representation, and the unstructured data alignment module uses a CLIP model to align text features and image features in the feature representation to obtain aligned unstructured data; the pre-trained Swin Transformer is pre-trained using the public COCO dataset.

[0094] S4: Apply the aligned structured data and the aligned unstructured data to the base large model to complete the construction of the multimodal large model.

[0095] Example 2

[0096] In this example, we give examples of structured data and unstructured data;

[0097] Structured Data Data Preprocessing: The data structure of the structured fine-tuning large model is usually similar to the format of the NL2SQL dataset and contains the following key parts so that the model can effectively generate SQL queries from natural language: 1. Input: Natural Language Query: This is the question asked by the user in natural language, which the model will use as input. Database Schema: It includes table structure information related to the query, usually including the name of the table, the name and type of the column, etc. This information helps the model understand the context of the data. 2. Output: SQL Query: The output goal of the model is to generate a SQL statement corresponding to the natural language query. This SQL statement should be directly executable on the database to obtain the results expected by the user. 3. Labels (Labels / Annotations): In supervised learning, the training data is usually labeled, that is, the correct SQL query statement. During the fine-tuning process, the model calculates the loss (for example, through a sequence-to-sequence loss function) by comparing the generated SQL with the correct labeled SQL, and gradually optimizes the ability to generate SQL. 4. Database content (Optional): Some fine-tuning datasets may also include part of the database content, especially when the model needs to generate SQL based on specific data. This may include sample data rows or specific query results to help the model understand the context.

[0098] Structured data fine-tuning: 1. The data format of structured pre-training data is:

[0099] {

[0100] "question": "How many states have a population greater than 10million?",

[0101] "schema": {

[0102] "table_name": "state_population",

[0103] "columns": [

[0104] {"column_name": "State", "data_type": "text"},

[0105] {"column_name": "Population", "data_type": "integer"} ]

[0107] },

[0108] "sql": "SELECT COUNT(*) FROM state_population WHERE Population>10000000"

[0109] }

[0110] In this embodiment, the multimodal large model is fine-tuned as follows: 1. Use the preprocessed data to fine-tune the model and gradually adjust the model parameters to make the generated SQL statements as accurate as possible. 2. Evaluation and optimization: Evaluate the model performance by comparing the generated SQL with the label SQL, and improve the model performance through optimization steps (such as adjusting the learning rate, model architecture, etc.). Figure 3 As shown, the specific fine-tuning process uses a graphical interface. 3. The fine-tuning method can be selected from Lora, Freeze, etc. 4. The selectable base models include but are not limited to Baichuan, Qwen, etc. Figure 4 The figure shows the training results of structured data in a multimodal large model. The input question is: "How many states have a population greater than 10 million?", and the model AI answers "There are over 88 countries woridwide with populations exceeding 10 million people.".

[0111] In this embodiment, we take pictures as an example of unstructured data. The first step is to upload Figure 5 As shown in the picture, the second step: Figure 5 As shown in the figure, if you input the question "Describe the number of infested trees in the above image, mark them with red dots and return the red dots with annotations", the model will analyze the content of the image and the model AI will answer "The number of infested trees in the above image is 18" and return the following answer: Figure 6 Shown is a picture of 18 infested trees marked with red dots.

[0112] Example 3

[0113] The present embodiment provides a device for constructing a multimodal large model based on low code, including: a data acquisition unit: used to acquire multimodal data; a first construction unit: used to construct a structured data processing module, the structured data processing module includes a table merging module, a model process construction module and a structured data alignment module, the structured data is subjected to a table merging operation by the table merging module to obtain a feature wide table, the feature wide table is subjected to the model process construction module to obtain a decision result of the feature wide table, the decision result is input together with the feature attributes in the feature wide table into the structured data alignment module to obtain aligned structured data; a second construction unit: used to construct an unstructured data processing module, the unstructured data processing module includes a target detection module and an unstructured data alignment module, the unstructured data is subjected to feature extraction by the target detection module to obtain a feature representation of the unstructured data, the feature representation is input into the unstructured data alignment module to obtain aligned unstructured data; a third construction unit: used to apply the aligned structured data and the aligned unstructured data to the base large model to complete the construction of the multimodal large model.

[0114] Example 4

[0115] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned low-code-based multimodal large model construction method when executing the program.

[0116] Example 5

[0117] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned low-code-based multimodal large model construction method are implemented.

[0118] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for building a multimodal large model based on low code, characterized in that: The following steps are involved: S1: Acquire multimodal data, where the multimodal data includes structured data and unstructured data; S2: constructing a structured data processing module, the structured data processing module includes a table merging module, a model flow building module and a structured data alignment module, the structured data is subjected to a table merging operation by the table merging module to obtain a feature wide table, the feature wide table is subjected to the model flow building module to obtain a decision result of the feature wide table, the decision result is input together with the feature attributes in the feature wide table into the structured data alignment module to obtain aligned structured data; The table merging module adopts PySpark table merging technology, takes PySpark as a visualization operator of the low-code platform, performs SQL-based visualization editing on the operator, completes the table merging operation, and obtains a feature wide table, which specifically includes: PySpark visualization: PySpark operator visualization technology is used to pre-process the structured data. PySpark operators provide a unified DataFrame API interface to implement table merging operations. Multiple tables are combined into a new structured data table through visualization SQL and PySpark to obtain a feature wide table. In the process implementation of the PySpark operator, the union operation is used to merge tables. Two or more data sets are merged into a new data set through the union operation. The data set is in the form of RDD or DataFrame. The model process building module specifically includes: The AutoML process is constructed by means of automated feature engineering, wherein the automated feature engineering includes an automated feature combination operator, wherein the feature wide table is subjected to feature combination analysis by the automated feature combination operator to generate features generated based on the combination method, and the importance of the features generated based on the combination method is evaluated by the automated feature combination operator, and finally the decision result of the feature wide table is obtained; The combination-based method is a combine function, and the automatic feature combination operator uses a bucketing technique; S3: constructing an unstructured data processing module, the unstructured data processing module comprising a target detection module and an unstructured data alignment module, the unstructured data undergoes feature extraction by the target detection module to obtain feature representation of the unstructured data, and the feature representation is input into the unstructured data alignment module to obtain aligned unstructured data; S4: Apply the aligned structured data and the aligned unstructured data to the base large model to complete the construction of the multimodal large model.

2. The method for building a multimodal large model based on low code according to claim 1 is characterized in that: In step S1, the structured data is a data table, and the unstructured data is a picture.

3. The method for constructing a multimodal large model based on low code according to claim 1 is characterized in that: The structured data alignment module in step S2 uses NL2SQL technology to convert the natural language in the structured data field into an executable structured query language in a relational database to obtain aligned structured data.

4. The method for building a multimodal large model based on low code according to claim 1 is characterized in that: The target detection module in step S3 uses a pre-trained Swin Transformer to extract features from the unstructured data to obtain the feature representation, and the unstructured data alignment module uses a CLIP model to align text features and image features in the feature representation to obtain aligned unstructured data.

5. The method for constructing a multimodal large model based on low code according to claim 4 is characterized in that: The pre-trained Swin Transformer is pre-trained using the public COCO dataset.

6. A device for constructing a multimodal large model based on low code, executing the method for constructing a multimodal large model based on low code as described in any one of claims 1 to 5, characterized in that: include: Data acquisition unit: used to acquire multimodal data; The first construction unit is used to construct a structured data processing module, which includes a table merging module, a model flow construction module and a structured data alignment module. The structured data is subjected to a table merging operation by the table merging module to obtain a feature wide table. The feature wide table is subjected to the model flow construction module to obtain a decision result of the feature wide table. The decision result is input into the structured data alignment module together with the feature attributes in the feature wide table to obtain aligned structured data. A second construction unit is used to construct an unstructured data processing module, wherein the unstructured data processing module includes a target detection module and an unstructured data alignment module. The unstructured data is subjected to feature extraction by the target detection module to obtain feature representation of the unstructured data, and the feature representation is input into the unstructured data alignment module to obtain aligned unstructured data. The third construction unit is used to apply the aligned structured data and the aligned unstructured data to the base large model to complete the construction of the multimodal large model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the low-code based multimodal large model building method as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the low-code based multimodal large model building method as described in any one of claims 1 to 5 are implemented.

Images