Method for solving enterprise data asset storage in multiple modes based on biological evolutionary algorithm

Abstract

The invention relates to the technical field of algorithms, in particular to a method for solving enterprise data asset storage in multiple modes based on a biological evolutionary algorithm, which comprises the following steps of: 1, receiving an enterprise data asset file in an Excel file format; 2, analyzing the Excel file through a data preprocessing module, and identifying a file format, a table structure and a field type; 3, processing the data by using a multi-modal large model, and selecting corresponding processing strategies according to different formats and table styles; 4, continuously optimizing the data processing model through a biological evolutionary algorithm to improve the accuracy and efficiency of data storage; and step 5, storing the processed data assets in a specified database system.The invention provides a multi-modal large model based on a biological evolutionary algorithm, and the multi-modal large model is used for solving various complex problems in enterprise data asset warehousing.

Description

Technical Field

[0001] The present invention relates to the field of algorithm technology, and in particular to a method for solving enterprise data asset warehousing problems based on a biological evolution algorithm multimodality. Background Art

[0002] As we all know, with the advent of the big data era, the level of enterprise informatization has been continuously improved, and the amount of data has shown explosive growth, especially in the financial, operational, sales and other departments of the enterprise. Data storage mainly relies on spreadsheet tools such as Excel. Although Excel is a very popular and easy-to-use data management tool, due to its many structural complexities and format differences, traditional manual data processing methods are cumbersome and prone to errors. Therefore, there is an urgent need for a more efficient and automated way to store data assets. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] In view of the deficiencies of the prior art, the present invention provides a method for solving the enterprise data asset warehousing problem based on a multimodal biological evolution algorithm.

[0005] (II) Technical solution

[0006] To achieve the above purpose, the present invention provides the following technical solution: a method for warehousing enterprise data assets based on a multimodal biological evolution algorithm, comprising the following steps:

[0007] Step 1: Receive the enterprise data asset file in Excel format;

[0008] 1. The system can automatically obtain data directly from the file system, database or API interface, and supports obtaining data from different file formats (Excel, CSV, JSON, etc.);

[0009] 2. An automatic monitoring mechanism can be set up to capture new data source files in real time to avoid manual intervention;

[0010] Step 2: parse the Excel file through the data preprocessing module to identify the file format, table structure and field type;

[0011] 1. The system automatically detects null values, duplicate values, and error values ​​in the data through preset rules, and can automatically correct them;

[0012] 2. For Excel files of different formats, the system automatically identifies the table header and column data types, and automatically unifies the formats as needed. For example, the date column is automatically converted to a unified format, and the numeric column is formatted as a floating decimal, etc.;

[0013] Step 3: Use a multimodal large model to process the data and select corresponding processing strategies according to different formats and table styles;

[0014] 1. The system automatically recognizes different Excel formats and styles, including complex situations such as merged cells, hidden rows / columns, cross-table references, and can automatically restore the table structure;

[0015] 2. For irregular data columns (such as numbers, text, and date columns), the system will automatically perform standardization to ensure that all data conforms to the expected format;

[0016] 3. The system can also optimize data conversion rules through machine learning algorithms, realize adaptive data processing, and improve the accuracy and flexibility of data processing;

[0017] Step 4: Continuously optimize the data processing model through biological evolution algorithm to improve the accuracy and efficiency of data storage;

[0018] 1. The system automatically generates SQL statements for data storage based on table content through deep learning and data inference models, and supports automatic storage of processed data into the specified database;

[0019] 2. Through multimodal processing technology, the system can automatically identify multiple modal data in the table, perform intelligent processing and convert it into a form that adapts to the database structure;

[0020] Step 5: Store the processed data assets in the specified database system.

[0021] In order to improve the use effect, the present invention has been improved in that the multimodal large model in step three includes a language processing model, an image recognition model and a table parsing model, which can automatically select and process different types of data files.

[0022] In order to improve the use effect, the present invention has the following improvements: the biological evolution algorithm in step 4 is used to optimize the parameters in the data processing model, including recognition accuracy, processing speed, etc.

[0023] (III) Beneficial effects

[0024] Compared with the prior art, the present invention provides a method for warehousing enterprise data assets based on a multi-modal biological evolution algorithm, which has the following beneficial effects:

[0025] This method for solving the enterprise data asset storage problem based on biological evolution algorithm multimodality has the following advantages:

[0026] Multimodal optimization: Traditional single optimization methods may perform well under specific constraints, but they are difficult to cope with complex data features or dynamically changing environments. Biological evolution algorithms can effectively optimize a variety of different data storage strategies by simulating the process of natural selection and species evolution, adapting to different data formats, structures and change patterns, thereby improving the efficiency of data asset storage;

[0027] Strong adaptability: Compared with traditional algorithms, biological evolution algorithms have strong global search capabilities and can optimize complex warehousing requirements, avoid local optimal solutions, and improve overall warehousing efficiency;

[0028] Multi-objective optimization: The storage of enterprise data assets not only needs to consider efficiency issues, but also involves multi-dimensional goals such as security, accuracy, and data consistency. Biological evolution algorithms can optimize multiple goals at the same time, improving the quality of data assets while meeting the different needs of enterprises;

[0029] Diversified data processing: For the storage of heterogeneous data sources (structured, semi-structured, and unstructured data), traditional methods may require multiple independent modules to process them, while multimodal biological evolution algorithms can optimize the processing of different data types under one framework, improving the flexibility and uniformity of the system.

[0030] Ability to cope with complex environments: In actual enterprise applications, data sources and storage requirements often change. The adaptive characteristics of biological evolution algorithms can effectively cope with these changes. As data sources, formats, and requirements change, the algorithms can adjust themselves and maintain stable performance.

[0031] Strong anti-interference ability: The biological evolution algorithm has strong anti-interference ability. It can maintain good storage effect when the data quality is not high or there are abnormal situations, avoiding data loss or wrong storage;

[0032] Dynamic resource scheduling: The biological evolution algorithm can dynamically adjust the resource configuration of data storage (such as computing resources, storage resources, etc.). In the case of large-scale data volume, it can effectively avoid the waste of resources. Through intelligent scheduling, it can balance the system load and reduce resource bottlenecks.

[0033] Improve resource utilization: Traditional methods may ignore the optimal allocation of resources, while multimodal methods based on biological evolutionary algorithms can continuously adjust resource allocation strategies through adaptive evolution, thereby improving resource utilization efficiency and reducing costs;

[0034] Coping with the scale of big data: As the amount of enterprise data grows, traditional data asset storage methods often find it difficult to cope with large-scale data processing needs. However, biological evolution algorithms have good scalability, can support large-scale data processing, and can play their advantages in a distributed environment.

[0035] Easy to integrate and expand: The framework of biological evolution algorithm can be seamlessly integrated with other technologies (such as big data platform, cloud computing platform, etc.), supporting the needs of enterprises of different sizes, and has strong scalability and adaptability;

[0036] In the process of data storage, biological evolution algorithms can not only optimize the storage structure, but also discover problems in the data (such as duplications, errors, missing values, etc.) through the evolution process, thereby improving the quality of the data;

[0037] The multimodal method based on biological evolution algorithm can introduce more security mechanisms in the process of data storage, such as dynamic encryption, access control, etc., to enhance data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] See also Figure 1 , a method for solving the enterprise data asset storage problem based on biological evolution algorithm multimodality includes the following steps:

[0041] Step 1: Receive the enterprise data asset file in Excel format;

[0042] 1. The system can automatically obtain data directly from the file system, database or API interface, and supports obtaining data from different file formats (Excel, CSV, JSON, etc.);

[0043] 2. An automatic monitoring mechanism can be set up to capture new data source files in real time to avoid manual intervention;

[0044] Step 2: parse the Excel file through the data preprocessing module to identify the file format, table structure and field type;

[0045] 1. The system automatically detects null values, duplicate values, and error values ​​in the data through preset rules, and can automatically correct them;

[0046] 2. For Excel files of different formats, the system automatically identifies the table header and column data types, and automatically unifies the formats as needed. For example, the date column is automatically converted to a unified format, and the numeric column is formatted as a floating decimal, etc.;

[0047] Step 3: Use a multimodal large model to process the data and select corresponding processing strategies according to different formats and table styles;

[0048] 1. The system automatically recognizes different Excel formats and styles, including complex situations such as merged cells, hidden rows / columns, cross-table references, and can automatically restore the table structure;

[0049] 2. For irregular data columns (such as numbers, text, and date columns), the system will automatically perform standardization to ensure that all data conforms to the expected format;

[0050] 3. The system can also optimize data conversion rules through machine learning algorithms, realize adaptive data processing, and improve the accuracy and flexibility of data processing;

[0051] Step 4: Continuously optimize the data processing model through biological evolution algorithm to improve the accuracy and efficiency of data storage;

[0052] 1. The system automatically generates SQL statements for data storage based on table content through deep learning and data inference models, and supports automatic storage of processed data into the specified database;

[0053] 2. Through multimodal processing technology, the system can automatically identify multiple modal data in the table, perform intelligent processing and convert it into a form that adapts to the database structure;

[0054] Step 5: Store the processed data assets in the specified database system.

[0055] The multimodal large model in step three includes a language processing model, an image recognition model and a table parsing model, and can automatically select and process different types of data files.

[0056] The biological evolution algorithm in step 4 is used to optimize the parameters in the data processing model, including recognition accuracy, processing speed, etc.

[0057] In summary, in the data access and analysis stage, the present invention uses a multimodal model to support data access in different formats, especially for the processing of Excel files, covering multiple details such as header recognition, content area positioning, row and column data analysis, etc.

[0058] File system access: supports loading various Excel formats (xls, xlsx, xlsm, etc.) files from the local file system;

[0059] Cloud storage access: supports reading Excel files from cloud storage (such as AWS S3, Azure Blob Storage, etc.) and realizes automatic loading through API interface;

[0060] API interface access: For some structured data sources, you can directly obtain Excel files through the API interface for parsing and processing;

[0061] Header recognition is a key step in data processing. Correctly identifying the header is crucial for subsequent data analysis. The header not only helps define the data field, but also helps the system understand the structure of the data. The modal model first identifies the header area in the file through a text parsing algorithm. The header is usually located in the first row or the first few rows of the table. The model can automatically detect the table structure and locate the header row. The system automatically classifies according to the text content in the header and identifies the actual meaning of each column (such as date, value, text, ID, etc.). For headers containing special symbols (such as timestamps, codes, etc.), the system can make adaptive adjustments to ensure correct classification. The following are several commonly used header recognition and positioning methods:

[0062] Fixed row header: If the header is located in a fixed row (such as the first row), it can be located by rule matching. For example, check whether the first row of the table contains a string or symbol to determine whether it is a header.

[0063] Column value type check: Based on the value type in the column (such as date, number, text, etc.), it is inferred that the table header may be located. For example, the value column may be followed by a data row, and the date column may belong to the table header field.

[0064] Text classification model: Use text classification algorithms and feature extraction technologies such as TF-IDF and Word2Vec to distinguish header rows from data rows;

[0065] Natural Language Processing (NLP) model: Use pre-trained language models to understand the semantics of header text and locate it based on context information. This method is particularly effective for headers containing unstructured text.

[0066] OCR (Optical Character Recognition) technology: For scanned image formats Excel (such as scanned PDF to Excel), OCR technology can be applied to recognize the text of the table header. OCR models (such as Tesseract) can identify the location of text, especially in the scanned documents of tables, to help identify the row where the table header is located;

[0067] Table structure analysis: By analyzing the row and column distribution, borders, and cell structure of the table, we can infer which rows may contain header information. For example, header rows usually have different formats (such as bold font, center alignment, etc.);

[0068] Graph Convolutional Network (GCN): If the position of the table header is not obvious, the model based on the graph convolutional network can automatically identify the table header by learning the graph structure of the table. This method is suitable for complex table structures, especially for tables that span rows or merge cells.

[0069] The data area is the part of the table that actually stores data. It is necessary to accurately identify the data area from the entire Excel file for subsequent parsing and processing. The following are common methods:

[0070] Empty row and column detection: By checking the empty rows and columns in the table and excluding invalid areas, you can identify which areas contain valid data (non-empty rows and columns) by scanning row by row and column by column.

[0071] Table structure analysis: By analyzing the boundaries of the table, determine which areas are distinguished as data areas by borders or background colors. For example, the valid data area of ​​the table is usually distinguished by some form of lines, background colors or borders.

[0072] Density analysis: Determine which areas contain valid data by counting the density of the data (such as the proportion of non-empty cells in a column of data). Valid data areas usually have a higher proportion of non-empty cells.

[0073] Row and column length detection: If the data in a row or column is long and uninterrupted, it may mean that the row or column is the boundary of the data area.

[0074] Convolutional Neural Network (CNN): Convolutional neural network is used to analyze table images and locate data areas. CNN can learn the table structure in the image and accurately identify the boundaries of the data area.

[0075] Object detection algorithms (such as YOLO): These algorithms can identify different areas in Excel table images, such as header areas, data areas, blank areas, etc., and accurately locate data areas.

[0076] Long Short-Term Memory (LSTM) network: It can be used to process the column data sequence of a table, learn the relationship between columns, and infer which columns belong to the data region. Especially when the table is dynamically changing, LSTM can effectively handle the location of data regions across rows.

[0077] Data parsing is the process of converting raw Excel data into a structured format. Usually, we need to ensure that the data type is correct, the format is uniform, and there is no redundant or erroneous data. The following are several commonly used parsing methods:

[0078] Date and time parsing: Use preset date formats (such as yyyy-MM-dd, MM / dd / yyyy, etc.) to parse date columns. If the date format in Excel is inconsistent, the system can automatically detect and convert it to the standard format;

[0079] Numeric column parsing: The system determines whether the column is a numeric type based on the column content (such as whether it contains letters or currency symbols). If the column contains currency symbols or thousandths symbols, the system removes the symbols and converts them to floating decimals or integers.

[0080] Data type classification model: Use classification models (such as SVM, decision tree, random forest, etc.) to identify the type of each column of data. When training the model, a large amount of labeled example data (such as date column, numeric column, text column, etc.) is used to let the system learn how to distinguish different data types;

[0081] Clustering algorithm: Clustering algorithms (such as K-means, DBSCAN, etc.) are used to cluster data to help identify columns of similar data types. For example, numeric columns, text columns, date columns, etc. may be clustered into different clusters and analyzed accordingly.

[0082] Combined parsing model: Combines rule-based parsing with machine learning models to improve flexibility and accuracy in parsing data columns. For example, for columns containing currency and percentage symbols, first use rules for basic parsing, and then use machine learning models to determine whether these columns need further processing.

[0083] Deep Neural Network (DNN): can be used to identify complex data types (such as nested tables, image data, etc.). DNN can handle complex relationships in data and automatically infer the type of data columns and convert them by training the network;

[0084] Pre-trained models such as BERT and GPT: If the data in the table has complex semantics (such as columns with long text), you can use pre-trained language models to perform deep semantic analysis to ensure the parsing accuracy of the text column;

[0085] Pivot Table is a common data summary and analysis tool in Excel. Processing Pivot Table requires special algorithms to extract useful data. Common methods are:

[0086] Relationship between table header and data row: Pivot table usually contains some summary fields (such as total, average, etc.), which have obvious relationship between table header and data area. The system can determine the structure of pivot table and parse it through pattern recognition;

[0087] Pattern recognition algorithms: They learn the common structure of pivot tables by training models, identify the dimension fields of pivot tables (such as rows, columns, summary values, etc.), and extract the actual data in the pivot tables. Such models usually combine convolutional neural networks (CNNs) and LSTM networks to capture the complex patterns of pivot tables.

[0088] Reverse analysis of pivot tables: Restore the data relationship in pivot tables through reverse reasoning methods (such as linear regression, machine learning, etc.). The system can reconstruct standard tables based on the measure values ​​and dimension information of pivot tables.

[0089] The system automatically identifies data types in Excel, including dates, text, numbers, Boolean values, etc., and automatically converts them into database-compatible data types based on preset rules or through learning mode. Especially for Excel with complex data types (such as numeric columns containing currency symbols), the system automatically removes non-numeric characters and converts them into numeric data.

[0090] The data cleaning and preprocessing stage is the basis of data analysis and modeling. Its core goal is to use a series of technical means to eliminate invalid, redundant, and erroneous data and improve the quality of data so that subsequent data analysis, model training, prediction and other links can proceed smoothly.

[0091] Missing values ​​are a common problem in data sets. Missing values ​​may be caused by a variety of reasons (such as data collection errors, input omissions, etc.). If not handled properly, it will affect the accuracy of model training and data analysis. Therefore, missing value processing is a very critical step in the data cleaning stage. Key technical solutions include:

[0092] Single missing deletion: If a row or column has a high percentage of missing values, delete the row or column. This method is suitable for situations where there is little missing data and deletion will not affect the integrity of the data set.

[0093] Delete records with multiple missing values: For records with multiple columns that are missing, if the number of missing values ​​is large, you can also choose to delete them directly;

[0094] Mean and median filling: For numerical data, use the mean or median to fill missing values. The median is often used to combat the impact of outliers.

[0095] Mode filling: For categorical data, the mode (the category with the highest frequency) is used to fill missing values;

[0096] Interpolation method: Applicable to time series data, interpolation and filling through adjacent values, such as linear interpolation, spline interpolation, etc.

[0097] Model-based filling: For example, using regression models, KNN (K-Nearest Neighbor) and other machine learning methods to predict missing values ​​and fill them. It is a common practice to infer missing values ​​based on similar records.

[0098] Duplicate values ​​are usually introduced due to improper data collection, storage or processing. Duplicate data can lead to deviations in analysis results, especially in statistical analysis and model training, which may cause information redundancy and overfitting. Therefore, deduplication processing is crucial. Key technical solutions include:

[0099] Row-based deduplication: Simple deduplication, by comparing the fields of each row of data, delete the identical rows. For completely duplicate records, the "deduplication" operation is generally used to delete them;

[0100] Column-based deduplication: Deduplication based on primary key. If there is a unique identifier (such as ID) in the data table, deduplication is performed using the primary key, retaining unique records and deleting other duplicate records.

[0101] Content-based deduplication: Fuzzy deduplication uses string matching techniques (such as Levenshtein distance and Jaccard similarity) to perform fuzzy matching for records with slight differences (such as spelling errors or different formats) and merge duplicate records;

[0102] Outliers refer to extreme values ​​that appear unreasonable in a data set. If outliers are not properly handled, they may affect the accuracy of statistical analysis and model prediction. Therefore, outlier processing is an important part of data cleaning. Key technical solutions include:

[0103] Box plot method (IQR method): by calculating the quartiles of the data, outliers outside the range of 1.5 times the IQR (interquartile range) of the upper and lower quartiles (Q1, Q3) are identified;

[0104] Z-score method: Calculate the Z-score of each data point in the data set. Points with Z-score values ​​exceeding a certain threshold (usually ±3) are considered outliers.

[0105] Clustering algorithms: Use clustering algorithms (such as K-means, DBSCAN) to identify low-density outliers, which are far away from other data points and are usually considered outliers;

[0106] Isolation Forest: A tree-based outlier detection algorithm that can efficiently identify isolated data points.

[0107] Box plots and scatter plots: Use visualization methods such as box plots and scatter plots to quickly identify outliers in the data;

[0108] Data format standardization is the process of ensuring that input data from different sources can be in a unified format. In an enterprise, data usually comes from different systems and may exist in various formats (different date formats, different numerical formats, etc.). Inconsistent data formats will lead to difficulties in subsequent processing. The purpose of the standardization process is to unify the data format and ensure the uniformity and consistency of the data. Key technical solutions:

[0109] Unify the date format: Use regular expressions or date parsing libraries (such as Python's dateutil and pandas) to unify the date format, for example, convert it to yyyy-MM-dd;

[0110] Time range unification: For timestamp data, unify it into a standard time format, such as converting timestamps in different formats into UTC time;

[0111] Unify decimal points and thousandths: Numeric fields may contain different separators (such as thousandths, currency symbols, etc.). Standardize the numeric format, remove invalid symbols, and unify them into floating decimals or integers;

[0112] Remove redundant characters from text: remove extra spaces, punctuation marks or special characters;

[0113] Unify case: Unify all text into lowercase or uppercase to avoid duplication caused by different case.

[0114] In the data conversion and optimization stage, the main function of this stage is to convert various raw data into a structured and standardized form suitable for storage, analysis and modeling. This process involves data format conversion, data cleaning, deduplication, optimization and other operations to ensure that subsequent analysis and reasoning can be carried out efficiently and accurately; the main task is to convert data into a usable and standardized format to facilitate subsequent analysis and modeling. This stage aims to improve data quality, ensure data consistency, optimize storage and computing efficiency, and deal with the fusion and conversion of multi-source data. Specifically, this stage includes data type conversion, data table merging and deduplication, biological evolution algorithm optimization and multimodal data fusion, which will be detailed below. Describe in detail the role of these stages, the technical solutions involved and their advantages and disadvantages, and analyze the feasibility of each solution. The current core role is to ensure data quality: the operations involved in this stage are aimed at fixing problems that may not have been fully resolved in the previous cleaning stage, improving the accuracy and consistency of the data, and improving data availability: through standardization and formatting, the data can be adapted to different analysis tools, algorithms and database storage, and optimizing data structure: by optimizing the data, it can be more suitable for subsequent analysis or machine learning modeling, reduce redundancy, improve model performance, and improve computing efficiency: data optimization includes operations such as compression and type conversion, which are aimed at improving the efficiency of subsequent calculations, especially for large data volumes;

[0115] Data type conversion is the process of converting data from one format to another to ensure that the data meets different analysis or system requirements. For example, converting numeric data to a string, or converting a date string to a date type. Correct data type conversion is the basis for data consistency, correctness, and integrity, especially for cross-platform or cross-system data exchange.

[0116] Automatic database mapping mainly refers to automatically mapping source data (such as data in Excel, CSV, JSON and other formats) to the corresponding table structure in the target database (such as MySQL, PostgreSQL, etc.) in a programmatic way. This process ensures the correct mapping between the source data and the target database structure, reduces manual intervention, and improves the efficiency and accuracy of data processing. The key technical solutions include:

[0117] Rule-based mapping engine:

[0118] Function: Preset rules to map fields in source data, usually by creating a mapping table or defining static rules. These rules match based on fixed field names, data types, or field attributes.

[0119] Machine learning automatic mapping:

[0120] Function: Use machine learning models (such as decision trees, KNN, SVM, etc.) to automatically learn data mapping rules based on historical data and labeled training sets. The model automatically infers the mapping rules between fields by learning the relationship between source data and target data.

[0121] Deep learning (such as Transformer, BERT, etc.) performs automatic mapping:

[0122] Function: Use deep learning models, especially natural language processing models such as Transformer and BERT, to automatically identify and map field relationships between different data sources. Such models are particularly suitable for mapping between complex text data and structured data.

[0123] Tool automatic mapping:

[0124] Function: ETL (Extract, Transform, Load) tools (such as Apache NiFi, Talend, Informatica, etc.) can automatically extract data from different data sources (such as Excel, CSV, API interfaces, etc.), transform it, and then load it into the target database;

[0125] Database modeling tools automatically map:

[0126] Function: Database modeling tools (such as Liquibase, Flyway, etc.) can be used to map database table structures and field types to the target database, and automatically generate SQL scripts for deployment;

[0127] Data type conversion in Python (such as pandas):

[0128] Function: Python's pandas library provides a wealth of data type conversion functions and can handle a variety of data formats, such as dates, numbers, text, etc. For CSV, Excel and other files, you can use pandas' read_csv(), read_excel() and other functions to automatically identify and convert data types;

[0129] Data type conversion in Java (such as JDBC):

[0130] Function: Java can map Java objects to data in the database through JDBC (Java Database Connectivity), and supports automatic conversion of common data types (such as String, Date, Integer, etc.);

[0131] Data conversion in JavaScript (such as Date.parse()):

[0132] Function: JavaScript's built-in conversion functions, such as Date.parse(), can convert date strings to Date objects. Methods such as Number() and String() can also handle data type conversions.

[0133] The conversion functions that come with databases and programming languages ​​can provide strong support in the data type conversion process. These built-in functions can quickly and accurately convert different data formats or data types and maintain the correctness of the data. Through these functions, common data type conversions (such as number, date, string conversion, etc.) can be quickly completed, reducing the development workload. By directly implementing conversions at the database query or code level, the error rate can be reduced. The key technical solutions include:

[0134] SQL built-in conversion functions:

[0135] Function: Most relational databases (such as MySQL, PostgreSQL, and Oracle) provide built-in conversion functions. For example, the CAST() and CONVERT() functions in MySQL can convert data from one type to another, and DATE_FORMAT() can format the date and time into a specified format.

[0136] Data table merging and deduplication is the process of merging data tables from different sources or removing duplicate values. This process is crucial for data integration and cleaning. It can ensure data consistency and accuracy and avoid the impact of redundant data. Key technical solutions include:

[0137] SQL-based deduplication and merging:

[0138] Function: Use JOIN operations (such as INNER JOIN, LEFT JOIN, etc.) in SQL queries to merge tables, use DISTINCT operations to remove duplicate rows, or use GROUP BY to aggregate data;

[0139] Processing data table merging and deduplication based on Python (pandas):

[0140] Function: pandas provides functions such as merge(), concat(), and drop_duplicates(), which can realize efficient data table merging and deduplication in memory;

[0141] Biological evolution algorithms (such as genetic algorithms) can be used to optimize certain steps in the data processing process, such as feature selection and hyperparameter tuning. In the data conversion and optimization stage, biological evolution algorithms can find the optimal data processing strategy or algorithm parameters by simulating the natural selection process, thereby improving the effect of data conversion and optimization. Key technical solutions include:

[0142] Genetic Algorithm (GA) Optimization:

[0143] Genetic algorithm is a heuristic search algorithm inspired by natural selection and genetics. It searches for the optimal solution in the solution space of the problem by simulating the evolution of biological species, including operations such as selection, crossover, and mutation. Genetic algorithm is often used to solve problems such as feature selection, data preprocessing, hyperparameter tuning, and combinatorial optimization. Its basic process includes:

[0144] Selection: Select suitable individuals (solutions) from the population as parents based on their fitness;

[0145] Crossover: Generate new individuals (offspring) through gene recombination, simulating the gene crossover of organisms;

[0146] Mutation: Randomly make small changes to the genes of individuals to increase the diversity of the search space and avoid falling into the local optimum;

[0147] Genetic algorithms are suitable for dealing with large-scale and complex optimization problems, especially those that cannot be effectively solved by traditional mathematical methods. For example, genetic algorithms can be used to obtain better solutions in feature selection, model hyperparameter tuning, or combinatorial optimization problems (such as the traveling salesman problem) in machine learning.

[0148] Ant colony algorithm and particle swarm algorithm optimization:

[0149] The ant colony algorithm imitates the foraging behavior of ants. It solves the shortest path problem by having artificial ants "trek" in the solution space and gradually optimize the path. The ant colony algorithm uses the concept of "pheromone". Ants release pheromones on the path, and other ants tend to choose the path with higher pheromone concentration, so that they can quickly converge to the optimal solution. The ant colony algorithm is often used to solve optimization problems, such as path planning, scheduling problems, and resource allocation.

[0150] Ant colony algorithm is particularly suitable for solving large-scale and complex combinatorial optimization problems. It is widely used in logistics distribution, path planning, production scheduling and other fields. It is very suitable for processing tasks that require finding the optimal solution in multi-dimensional space.

[0151] The core purpose of the multimodal data fusion stage is to effectively integrate data from different modalities (such as text, images, audio, video, etc.) for unified analysis, modeling, and reasoning. The key challenge of this stage is how to transform these heterogeneous data into a consistent and standardized form and resolve possible differences and mismatches between modalities.

[0152] The role of the multimodal data fusion stage:

[0153] Data integration: Data sources of different modalities usually have different structures and forms. Text data may be structured or unstructured, while image and video data are usually high-dimensional multi-channel data. The role of multimodal data fusion is to merge data from these different sources in a unified way to build a unified model or data framework that can simultaneously process text, image, audio and other information;

[0154] Enhanced data representation capabilities: Single-modal data often cannot fully reflect the multi-dimensional information of the complex real world. By integrating data from multiple modalities, problems can be analyzed from multiple angles and levels to capture more information. For example, images and text can complement each other. Images provide intuitive visual information, while text can provide more detailed contextual descriptions, thereby obtaining a more comprehensive understanding.

[0155] Improve the accuracy of analysis and reasoning: In many applications, such as intelligent question answering, recommendation systems, and sentiment analysis, by combining multimodal data, the model can make more accurate judgments by integrating information from each modality. For example, in a video analysis system, combining the visual information of the video and the speech content in the audio can more accurately identify events or scenes in the video.

[0156] In practical applications, enterprises or systems often face data from multiple sources, formats, and scales. The standardization and integration of these data are the key to enabling subsequent analysis and decision-making to proceed smoothly.

[0157] Data preprocessing and standardization:

[0158] 1. Clean and preprocess data from different modalities to remove noise and redundancy. For example, text data may need to remove stop words and punctuation marks, and image data may need to remove background noise or perform deblurring.

[0159] 2. Convert data of different modalities into a unified format or standard, such as converting image data into vector representation, text data into word vectors (such as Word2Vec, BERT, etc.), audio data into Mel-spectrograms, etc.;

[0160] Feature extraction and representation learning:

[0161] 1. Text data: Use natural language processing technology (such as BERT, GPT and other pre-trained language models) to extract the semantic features of the text.

[0162] 2. Image data: extract visual features of the image through convolutional neural networks (CNN), or process it through pre-trained visual models (such as ResNet, VGG, Transformer, etc.),

[0163] 3. Audio data: The audio data is converted into feature vectors that can be fused with other modalities through acoustic feature extraction methods (such as MFCC, STFT, etc.).

[0164] 4. These feature vectors can provide a unified representation space for subsequent fusion.

[0165] Feature fusion method:

[0166] 1. Early Fusion: The original features from different modalities are directly spliced ​​or combined. Data fusion is performed at this stage. For example, the feature vector of the image and the feature vector of the text are directly connected for modeling.

[0167] 2. Mid Fusion: Perform independent feature extraction and modeling on data of different modalities, and perform information fusion at the intermediate level. For example, CNN can be used to process images and RNN can be used to process texts, and feature fusion can be performed at the intermediate level of the model.

[0168] 3. Late Fusion: Train a model for each modality separately, and finally fuse the prediction results of each modality. For example, the results of different models can be fused by weighted voting, weighted averaging, maximum voting, etc.

[0169] Cross-modal learning and alignment:

[0170] 1. In multimodal data fusion, data from different modalities sometimes exist in different semantic spaces, so their semantic representations need to be aligned. For example, the content of an image and the text describing the image need to be aligned in the same space so that they can complement each other. Common methods include **Cross-Modal Alignment** technology, such as training images and texts in a shared embedding space so that images and texts can have similar representations in this space.

[0171] 2. Common models such as CLIP (Contrastive Language-Image Pretraining) use contrastive learning methods to train shared representations of images and texts.

[0172] Deep Learning Models:

[0173] 1. Transformer architecture: Due to its powerful ability to process long sequence data, Transformer is widely used in multimodal data fusion tasks. For example, VisualBERT and VL-BERT models combine image and text information and use Transformer structure for multimodal learning.

[0174] 2. Dual-Encoder Network: This type of network encodes the data of each modality separately and fuses them at the output layer. It is suitable for situations where the information is not completely matched (such as the relationship between text and images).

[0175] 3. Multimodal Neural Networks: By designing a multimodal neural network architecture, data from multiple modalities can be processed in a unified model to achieve joint learning and optimization of features.

[0176] Optimization and Adaptive Learning:

[0177] 1. In the process of multimodal data fusion, biological evolution algorithms, genetic algorithms and other methods are usually used to optimize the parameters of the model to improve the fusion effect and reasoning accuracy.

[0178] Through intelligent algorithms, the data weights of different modalities can be continuously optimized to ensure that the information of each modality can effectively contribute to the final model.

[0179] In the data storage and reasoning stage, data processing not only includes importing the processed data into the database, but also involves real-time reasoning and analysis of the data to support subsequent business decisions or model training. The key task of this stage is to ensure that the data can be quickly and accurately stored in the target storage system and that efficient data analysis and prediction can be performed through the reasoning engine;

[0180] In the data storage stage, the most critical thing is to automatically convert the processed data into SQL statements suitable for the target database to avoid manual errors and improve efficiency. This process should be able to automatically generate appropriate SQL statements based on the data source type, database table structure and constraints;

[0181] Data Mapping and Conversion:

[0182] 1. Ensure that the system can identify the mapping relationship between the source data and the target database. For example, when data is imported from CSV, JSON or other formats, the system should be able to automatically identify the field type and map it to the appropriate type of the target database (such as integer, date, string, etc.).

[0183] 2. Use data conversion models (such as ETL tools) to handle data format conversion and ensure that the data conforms to the format requirements of the target database.

[0184] SQL statement generation:

[0185] The system generates SQL statements based on the mapped data, which mainly include the following operations:

[0186] 1. Create a table: automatically generate CREATE TABLE statements based on the data structure, and automatically define constraints such as primary keys, foreign keys, and indexes.

[0187] 2. Data insertion: Generate INSERT INTO statements and support batch insertion. For large amounts of data, batch insertion (INSERT INTO...VALUES...) can improve performance.

[0188] 3. Table structure optimization: Generate appropriate table indexes based on field type and data volume to improve query performance.

[0189] Technical implementation:

[0190] SQL template engine: Use Velocity, Freemarker and other template engines to dynamically generate SQL statements.

[0191] ORM framework: Using tools such as Hibernate and MyBatis, you can achieve automatic mapping of objects to database table structures and automatically generate corresponding SQL statements.

[0192] Different databases have different syntax, data types, constraints and other characteristics. Therefore, the system needs to be able to adapt to multiple databases and dynamically generate corresponding storage SQL according to the type of the target database;

[0193] Database type identification:

[0194] 1. The system automatically identifies the target database type (such as MySQL, PostgreSQL, MongoDB, etc.) and selects the appropriate SQL statement structure according to different types of databases.

[0195] 2. For relational databases (RDBMS), the main focus is on SQL syntax (such as creating tables, inserting data, etc.); for NoSQL databases, you need to consider the mapping of data structures (such as documents, collections, key-value pairs, etc.).

[0196] Database Adapter:

[0197] 1. Use the Adapter Pattern to create a separate adapter class for each database type. Each adapter class generates corresponding SQL statements or API calls based on the characteristics of the target database.

[0198] 2. For example, you can use JDBC connections to execute SQL statements for relational databases, and you can use database-specific APIs (such as Mongoose for MongoDB) for NoSQL databases.

[0199] Configuration and expansion:

[0200] 1. Configure database type and adaptation method, support dynamic expansion of new database types, and ensure that the system can support multiple databases through automatic detection of configuration files or database types.

[0201] Tool selection: Use a database connection pool (such as HikariCP) to efficiently manage database connections and improve system performance;

[0202] Data reasoning and verification aims to ensure that data complies with business rules and database constraints before being stored. The key to this stage is to check the consistency and integrity of data based on business requirements and data quality standards.

[0203] Data consistency check:

[0204] 1. Check whether the data meets the database constraints (such as primary key, foreign key, unique constraint, etc.),

[0205] 2. For data in specific business scenarios such as finance and orders, business rule verification is performed (such as whether the amount exceeds the budget, whether the timestamp is reasonable, etc.).

[0206] Reasoning model and algorithm:

[0207] 1. Use rule engines or machine learning models to verify data. For example, use rule engines such as Drools to define complex business rules and verify data.

[0208] 2. For some complex verification requirements, inference analysis can be performed based on machine learning models to predict whether the data conforms to certain trends or patterns.

[0209] Real-time data verification:

[0210] 1. When data is entered into the database, the data is checked through real-time reasoning, for example, checking the continuity of timestamps, verifying the matching of amount columns with budget data, and verifying the logical consistency of order data.

[0211] Anomaly detection and processing:

[0212] 1. Design anomaly detection mechanisms to identify abnormal patterns or illogical situations in data in real time. For example, use anomaly detection algorithms based on decision trees to automatically identify anomalies in data.

[0213] 2. For data that does not meet the requirements, an alarm can be triggered, the data can be corrected or pushed to manual review. Technical implementation:

[0214] 1. Use message queues such as Apache Kafka to implement asynchronous verification, asynchronous processing of real-time reasoning in large data volumes and ensure performance.

[0215] Use machine learning models (such as Random Forest, SVM, etc.) to learn historical data, train and verify models, and automatically identify data problems;

[0216] The goal of automated warehousing operations is to avoid errors in manual operations through automated processes and to significantly improve the efficiency of data warehousing. The system needs to achieve fully automated operations from data reception, cleaning to warehousing.

[0217] Automated Workflows:

[0218] 1. Build an automated data warehousing process, which involves data reception, cleaning, conversion, verification, reasoning, SQL generation, warehousing execution, etc. The entire process is scheduled and managed through workflow management tools such as Apache Airflow and Luigi.

[0219] 2. The data warehousing process includes batch data import and real-time data import. For batch data, use scheduled tasks and data streams to import in batches; for real-time data, use message queues (such as Kafka) to obtain and import in real time.

[0220] Automated execution:

[0221] 1. Use JDBC connection or database client tools (such as DBeaver, Navicat) to execute automatically generated SQL statements.

[0222] 2. For large-scale data, adopt batch and segment storage strategies to ensure the maximum performance of the database. Transaction control can be used to ensure the atomicity and consistency of data.

[0223] Fault tolerance and rollback mechanism:

[0224] 1. In the process of automated warehousing, a fault-tolerant mechanism is designed. Once an error occurs, the data is automatically rolled back to ensure the consistency of the database state. The database transaction mechanism is used to ensure the atomicity of the warehousing operation.

[0225] 2. Automatically retry the warehousing task in case of abnormal situations such as network or system crashes.

[0226] Performance optimization:

[0227] Design optimization strategies to meet the demand for large amounts of data to be stored, using methods such as batch insert, parallel processing, and partition tables to improve data storage performance;

[0228] This application is aimed at the implementation case integration of data asset storage. The specific operations are as follows:

[0229] 1. At the beginning of the development process, we first need to conduct a detailed investigation and analysis of the various formats and styles of current Excel files. After investigation, the team found that the formats of Excel files are mainly divided into the following categories:

[0230] Standard table: regular data structure, complete rows and columns, clear fields,

[0231] Tables with merged cells: Some cells in this table may be merged, which may affect the accuracy of data extraction.

[0232] Pivot table: The relationship between rows, columns and values ​​of data is relatively complex, and additional analysis of its structure is required.

[0233] Excel files with embedded images: For example, scanned Excel files, the data in the image needs to be extracted using OCR technology.

[0234] Through the study of these common Excel spreadsheet formats, we have identified the basic requirements for data processing: on the one hand, intelligent data recognition and extraction are required, and on the other hand, different strategies need to be adopted for different formats to ensure the accuracy of data processing.

[0235] 2. Based on the research phase, we developed a detailed technical plan and identified the following key parts:

[0236] Multimodal data access: The system is designed with a data access module that supports multiple data formats (xls, xlsx, csv, etc.), which can automatically identify file types and perform preprocessing.

[0237] Table structure analysis: An algorithm based on automatic header recognition and content area positioning is designed to ensure that the data area can be accurately located regardless of whether the table has merged cells or pivot tables.

[0238] Multimodal data processing: Using a multimodal large model, combined with text parsing, image recognition (OCR) and other technologies, to process data in various formats and complex tables to ensure accurate data extraction.

[0239] Biological evolution algorithm optimization: The biological evolution algorithm continuously optimizes the data processing strategy to improve processing efficiency and accuracy. Especially when facing complex tables, the algorithm can automatically adjust the processing strategy according to the specific situation.

[0240] Finally, the technical team divided the solution into the following implementation steps: data access, table analysis, data processing, data storage, result feedback and optimization.

[0241] 3. During the implementation process, the team first built a unified Excel file reading module using existing open source libraries (such as Apache POI, OpenXML SDK, etc.) to ensure support for files of different formats. Next, they developed the following dedicated parsing methods for different table structures in Excel:

[0242] Header recognition and content area positioning: Through the rule engine and machine learning algorithm, it can identify the header position and locate the data area.

[0243] Merged cell processing: In a table with merged cells, the system will adjust the row and column mapping of the data according to the range of the merged cells to ensure the integrity of the data.

[0244] Pivot table parsing: Through the special pivot table parsing module, it is possible to extract the rows, columns, and data values ​​in the pivot table and reorganize them into a standard table structure.

[0245] Image data processing: For Excel files with images, OCR technology is used to extract text data from the image and merge it with the table data.

[0246] During the implementation process, we also used biological evolution algorithms to continuously optimize the processing flow. By setting different initial parameters, conducting multiple experiments, and adjusting the optimization strategy based on the experimental results, we ensured that the system could make the best decision when processing different types of Excel files.

[0247] 4. After the initial implementation, we analyzed the actual processing results and found some problems and challenges:

[0248] Mixed data types in a table: Some columns may contain multiple data types such as text, numbers, dates, etc., which may cause recognition errors.

[0249] Merged cell recognition accuracy problem: In complex tables, the splitting and data positioning of merged cells may deviate, affecting the final result.

[0250] Pivot table data missing problem: For complex pivot tables, some data may be misjudged or lost.

[0251] To address these issues, we made adjustments to the system:

[0252] Multimodal optimization: By adjusting the parameters of the multimodal large model, it can better adapt to the recognition of different data types.

[0253] Algorithm fine-tuning: The biological evolution algorithm was fine-tuned, and the fitness function and mutation strategy were adjusted so that the algorithm can more accurately identify complex formats such as merged cells and pivot tables.

[0254] Data Validation: Added data validation steps, especially when processing key fields such as date and amount, added validation rules to ensure the accuracy of data.

[0255] 5. After adjustment, the system can more accurately process Excel files of different formats and styles. In actual application, the system can

[0256] Automatically recognize and parse standard tables, tables with merged cells, pivot tables, and Excel files with embedded images.

[0257] Improve the accuracy of data extraction through multimodal technology, especially in image processing and complex data analysis,

[0258] Use biological evolution algorithms to continuously optimize data processing processes and improve the overall efficiency and accuracy of the system.

[0259] Ultimately, the system can greatly reduce manual intervention, improve the efficiency of data entry, and ensure the integrity and accuracy of the data.

[0260] 6. After multiple rounds of optimization and adjustment, the final solution can effectively process Excel files of various formats and successfully automatically store data. The implementation results are as follows:

[0261] Improved data processing efficiency: Compared with manual processing, the system can complete the processing and storage of large quantities of data within minutes, saving a lot of time.

[0262] Improved data accuracy: Through accurate identification of multimodal models and optimization of biological evolution algorithms, the data processing error rate is reduced to near zero.

[0263] Support for multiple Excel formats and table styles: The system can automatically adapt to Excel files in multiple formats, including common standard tables, tables with merged cells, pivot tables, and files with embedded images.

[0264] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for warehousing enterprise data assets based on a multi-modal biological evolution algorithm, characterized by: The following steps are involved: Step 1: Receive the enterprise data asset file in Excel format; 1. The system can automatically obtain data directly from the file system, database or API interface, and supports obtaining data from different file formats (Excel, CSV, JSON, etc.); 2. An automatic monitoring mechanism can be set up to capture new data source files in real time to avoid manual intervention; Step 2: parse the Excel file through the data preprocessing module to identify the file format, table structure and field type; 1. The system automatically detects null values, duplicate values, and error values ​​in the data through preset rules, and can automatically correct them; 2. For Excel files of different formats, the system automatically identifies the table header and column data types, and automatically unifies the formats as needed. For example, the date column is automatically converted to a unified format, and the numeric column is formatted as a floating decimal, etc.; Step 3: Use a multimodal large model to process the data and select corresponding processing strategies according to different formats and table styles; 1. The system automatically recognizes different Excel formats and styles, including complex situations such as merged cells, hidden rows / columns, cross-table references, and can automatically restore the table structure; 2. For irregular data columns (such as numbers, text, and date columns), the system will automatically perform standardization to ensure that all data conforms to the expected format; 3. The system can also optimize data conversion rules through machine learning algorithms, realize adaptive data processing, and improve the accuracy and flexibility of data processing; Step 4: Continuously optimize the data processing model through biological evolution algorithm to improve the accuracy and efficiency of data storage; 1. The system automatically generates SQL statements for data storage based on table content through deep learning and data inference models, and supports automatic storage of processed data into the specified database; 2. Through multimodal processing technology, the system can automatically identify multiple modal data in the table, perform intelligent processing and convert it into a form that adapts to the database structure; Step 5: Store the processed data assets in the specified database system.

2. The method for warehousing enterprise data assets based on biological evolution algorithm multimodal solution according to claim 1 is characterized by: The multimodal large model in step three includes a language processing model, an image recognition model and a table parsing model, and can automatically select and process different types of data files.

3. The method for warehousing enterprise data assets based on biological evolution algorithm multimodal solution according to claim 2 is characterized by: The biological evolution algorithm in step 4 is used to optimize the parameters in the data processing model, including recognition accuracy, processing speed, etc.

Images